In a recent post on facebook, Dan Richards was wondering aloud about the realities of recruiting (study participants) versus “our rush to get the right amount of n=?” He was specifically referring to student recruitment where there is a great disparity in the regulations that govern this from institution to institution (something Dan himself pointed out) and the fact there is often so little inducement (like a gift card, bonus points, or my addition here, a simple thank you).
As someone who is finishing up a couple of studies that takes a meta-look at the scholarship in technical and professional communication (TPC), I remarked that I had a lot to say about this. Before I start things about the subject let me say that while I am widely read in composition/rhetoric, the comments below are specific to TPC. The data points are from a work in progress and are only in draft at this moment. Final caveat: As many of you know, I’ve been ranting for years about research study design, research questions, research methods, and research methodology for years (see five year’s worth of presentations at SCCI and my recent publication in TC and forthcoming edited collection…all of which is the start of a serious arc of research on research).
What the pedagogical research in the field looks like (for a five year snapshot) is
- 82 pedagogical studies out of 404 research articles
- 41 that actually use the classroom as the research site
- 41 are using what we’re tentatively calling exposition, which “promotes a certain pedagogical or theoretical framework is the integral point of the article. This framework is typically developed on other frameworks, and classroom-based research may or may not be utilized to help bolster the theoretical arguments.” (this is in quotes because it was smartly written by one of my collaborators. If we’re lucky, you may see it in print one day.)
There’s a wide range of empirical and textual methods in those 41 classroom based articles and only 10 or so are some sort of empirical based study. Why do I write “or so”? Well, in some cases it’s painfully not clear really what took place in the study, and more importantly to this post, it is unclear the exact involvement of students or their texts. This is problematic from a number of angles and I implore journal editors and reviewers to ask questions about the study design during peer review. Not only does it diminish the credibility of our research, it makes it near impossible to replicate (or to attempt similar studies that could complement or intersect) and it really calls into question issues of trustworthiness, reliability, and validity. I know, I know, there are many in TPC that feel these sorts of research terms and language are dirty words because that’s not the “type of study” I’m trying to do. That’s great. You’re entitled to your opinion, and I’m entitled to mine when I say that if a study doesn’t address these things in some manner, I’m not reading it and I sure as hell will not cite it or use it my classroom (except as a model of what not to do) nor will I take any of it’s findings seriously.
Some of the questions that have been raised about this pedagogical research in the field project directly intersect with Dan’s initial question/concern and with the concerns/comments that a number of commenters made.
Here are a few of them:
— how is recruitment effected by institutional regulations, access to students (your own or someone else’s), incentives or the value proposition to the student (which is often not great on any level) and more importantly, how can we mitigate some of these factors?
for years I’ve made the argument that research methods are ethics (an argument made simultaneously by Annette Markham from Internet studies) and my rationale for it is that the researcher’s ethical stance to the research problem/question is manifested in the method(s) selected. Mixed into this is the recruitment because that is intimately connected to the method (and then one step backward to the methodology). In any case, thinking through what we’re asking our participants to do is as much a methods and ethics question as anything else. Both of these topics need more attention and in doing so, we may begin to be able to answer questions about recruitment.
— are there a series of burning questions we would like answered through pedagogical research? Dan’s study is a replication. None of pedagogical research in our journal years is replications nor does it really build on anything, which raises the question about the field’s value of novelty.
TPC has a history of pedagogical research. Like it’s related field, composition studies, the first scholarship consistently published was on pedagogical and programmatic practices. We’re definitely to the point in our history that we need to fine tune our approaches to research questions around pedagogy and programs. And we need to be really careful in this question construction about what students can actually answer. there are a number of studies (and many more that I’ve reviewed over the last few years) that make grandiose claims about the practice of technical communication based on having their students do something. That’s problematic. As a field, we need to pay close attention to the questions we’re asking and then designing studies that can help answer those questions. Too often, it seems that we reverse engineer the study and what we ask of students to answer the question in a pre-determined way. Working as a collective around some of the pressing issues that we feel need answers to (like how successful is the service course in actually preparing students to write on the job?) could generate good questions, and more importantly, it could lead to more cross-institutional research. See next point.
— local studies are a great way to start, but how can we design more pedagogical centered studies that cross institutions to really get at what we’re doing in the classroom and how to make it better?
As someone who is working on a couple of cross-institutional studies around pedagogy/programs, I KNOW (with every fabric of being) how difficult it is to do this work, and to do it well. But, I would definitely argue that we have to do more of it. If only I had the time, I would offer to do Dan’s replication at my own place because that’s when we can really start to do some important field-wide work and it simultaneously can answer local problems. It’s mind boggling to me that in TPC we truly don’t practice some of the basic tenets of TPC that date back generations, and that is, collaborative work, in depth questions, precise research study design (which can be seen in the early modern era for goodness sake), and in our current technological state, the help of technologies to move all of these things along. I do get institutional pressures and documents that guide our lives (such as reappointment, promotion and tenure documents and annual review forms), but there comes a time that we (those of us tenured and in positions to do it) need to start pushing back and advocating for the type and kind of research we want to see, that the field needs, rather than just doing more of the same. I can tell you this: if I never see a survey request again, it will not be soon enough. (While this is a rant for another post, most surveys in the field are not gathering data that actually answer the question being posed and it distorts and contorts the primary usage of the survey instrument as a method.)
— what role do and should regulatory agencies and regulations play in research?
There was a point made about the way research in TPC often excludes itself from regulatory oversight because the researcher feels that the study is exempt (meaning is “not human subjects research”). Many of the classroom studies in our study made no indication of IRB or even any ethical considerations of researching within their own classrooms (which let me be clear, I personal find highly problematic). Every study in which you’re dealing with people needs to be submitted to the IRB. That’s just good research. The fact the published studies left so many questions unanswered about the study design, which should begin and end with ethical considerations, is something that the field seriously needs to consider and discuss. In short, as researchers, we have a major responsibility to ensure that we’ve given complete and due diligence to the ethical considerations of our studies–no matter what the IRB says. Part of this ethical consideration and ethical orientation is a deep and real concern in how this research recruitment may be understood by students.
Some of the comments on Dan’s post (thanks for the reminder, Jordynn) point to the fact that the IRB process can actually HELP by CLARIFYING and honing your research questions and the IRB (good ones) will also encourage deep ethical thinking that connects specifically to your research question and research participants. Yes, I’ve had studies altered by the IRB. But I’ve come to realize, to respect, and to learn from those decisions so there is in fact a study that I am working on that changed from the pilot to the full study based on the IRB. This wasn’t a bad thing. It was simply a thing, and part of participant driven research.
So that’s the start of some of the things I wanted to say from Dan’s post and his own research project and the pedagogical published research project that we’re in the latter stages of completion. One of the big things I wanted to say about Dan’s post was good for him in talking about these things out loud and good for the discussion that it generated. We need to have more of these sorts of discussions in print (especially) and in other avenues. We need these discussions because as a field, we have to do research better. To do that, starts with deliberation and reflection about our practices.