In popular writing on psychology, science, and education, we often encounter the phrase “research has shown.” Beware of it. Even its milder cousin, “research suggests,” may sneak up and put magic juice into your eyes, so that, upon opening them, you fall in love with the first findings you see (until you catch on to the trick).
Research rarely “shows” much, for starters—especially research on that “giddy thing” known as humanity.* Users of the phrase “research has shown” often commit any of these distortions: (a) disregarding the flaws of the research; (b) misinterpreting it; (c) exaggerating its implications; or (d)cloaking it in vague language. Sometimes they do this without intent of distorting, but the distortions remain.
Let’s take an example that shows all these distortions. Certain teaching methodologies emphasize a combination of gesture, speech, and listening. While such a combination makes sense, it is taken to extremes by Whole Brain Teaching, a rapid call-and-response pedagogical method that employs teacher-class and student-student dialogue in alternation. At the teacher’s command, students turn to their partners and “teach” a concept, speaking about it and making gestures that the partner mimics exactly. Watch the lesson on Aristotle’s “Four Causes,” and you may end up dizzy and bewildered; why would anyone choose to teach Aristotle in such a loud and frenzied manner?
The research page of the Whole Brain Teaching website had little research to offer a few months ago. Now it points to a few sources, including a Scientific American article that, according to the WBT website, describes “research supporting Whole Brain Teaching’s view that gestures are central to learning.” Here’s an instance of vague language (distortion d). Few would deny that gestures are helpful in teaching and learning. This does not mean that we should embrace compulsory, frenetic gesturing in the classroom, or that research supports it.
What does the Scientific American article say, in fact? There’s too much to take apart here, but this passage caught my eye: “Previous research has shown”—eek, research has shown!— “that students who are asked to gesture while talking about math problems are better at learning how to do them. This is true whether the students are told what gestures to make, or whether the gestures are spontaneous.” This looks like an instance of exaggerating the implications of research (distortion c); let’s take a look.
The word “told” in that passage links to the article “Making Children Gesture Brings Out Implicit Knowledge and Leads to Learning” by Sara C. Broaders, Susan Wagner Cook, Zachary Mitchell, and Susan Goldin-Meadow, published in the Journal of Experimental Psychology, vol. 136, no. 4 (2007), pp. 539–550. The abstract states that children become better at solving math problems when told to make gestures (relevant to the problems) during the process. Specifically, “children who were unable to solve the math problems often added new and correct problem-solving strategies, expressed only in gesture, to their repertoires.” Apparently, this progress persisted: “when these children were given instruction on the math problems later, they were more likely to succeed on the problems than children told not to gesture.” So, wait a second here. They didn’t have a control group? Let’s look at the article itself.
The experimenters conducted two studies. The first one involved 106 children in late third and early fourth grade, whom the experimenters tested individually. For the baseline set, children were asked to solve six problems of the type 6 + 3 + 7 = ___ + 7, without being given any instructions on gesturing. Children who solved any of the problems correctly were eliminated from the study at the outset. (Doesn’t this bias the study considerably? Shouldn’t this be mentioned in the abstract?)
From there, the students were assigned to groups for the “manipulation phase” of the study. Thirty-three students were told to gesture; 35 were told to keep their hands still; and 38 were told to explain how they solved the problems. The students who were told to gesture added significantly more “strategies” to their manipulation than did the students in the other two groups; however, nearly all of these strategies were expressed in gesture only and not in speech. Across the groups, students added a mean number of 0.34 strategies to their repertoire, 0.25 of which were correct (the strategies, that is, not the solutions).
It is not clear how many students actually gave correct answers to the problems during the manipulation phase. The study does not provide this information.
The second study involved 70 students in late third and early fourth grade; none had participated in the first study. After conducting the baseline experiment (where no students solved the problems correctly), the researchers divided the students into two groups for the manipulation phase. Children in one group were told to gesture; children in the other group were told not to gesture. The researchers chose these two groups because they were “maximally distinct in terms of strategies added.” (How did they know this in advance? This is not clear.)
Again, the students who had been told to gesture added more strategies to their repertoire; those told not to gesture added none. The researchers state later, in the “discussion” section of the paper: “Note that producing a correct strategy in gesture did not mean that the child solved the problems correctly. In fact, the children who expressed correct problem-solving strategies uniquely in gesture were, at that moment, not solving the problems correctly. But producing a correct strategy in gesture did seem to make the children more receptive to the later math lesson.”
After the children had solved and explained the problems in the manipulation phase, they were given a lesson on mathematical equivalence. (There was no such lesson in the first study.) The experimenter used a consistent gesture (moving a flat palm under the left side of the equation and then under the right side) for each of the problems presented. Then the students were given a post-test.
On the post-test, the students told not to gesture solved a mean of 2.2 problems correctly (out of six); those told to gesture solved a mean of 3.5 correctly. (I am estimating these figures from the bar graph.)
Why would anyone be impressed by the results? For some reason the researchers did not mention actual performance in the first study. In the second, it isn’t surprising that the students told not to gesture would fare worse on the test. A prohibition against gesturing could be highly distracting, as people tend to gesture naturally in one way or another. Again, there was no control group in the second study. Moreover, neither the overall mean performance on the test or the performance difference between the groups is particularly impressive, given that the problems all followed the same pattern and should have been easy for students who grasped the concept, provided they had their basic arithmetic down.
The researchers do not draw adequate attention to the two studies’ caveats or consider how these caveats might influence the conclusion (distortions a and b). In the “discussion” section of the paper, they state with confidence that “Children told to gesture were more likely to learn from instruction than were children told not to gesture.”
This is just one of myriad examples of research not showing what it claims to show or what others claim it shows. I have read research studies that gloss over their own gaps and weaknesses; popular articles that exaggerate the implications of this research; and practitioners who cite the popular articles in support of their particular method. When I hear the phrase “research has shown,” I immediately suspect that it isn’t so.
*From Shakespeare’s Much Ado About Nothing; thanks to Jamie Lorentzen for reminding me of the phrase.