Research Has Shown—Just What, Exactly?

In popular writing on psychology, science, and education, we often encounter the phrase “research has shown.” Beware of it. Even its milder cousin, “research suggests,” may sneak up and put magic juice into your eyes, so that, upon opening them, you fall in love with the first findings you see (until you catch on to the trick).

Research rarely “shows” much, for starters—especially research on that “giddy thing” known as humanity.* Users of the phrase “research has shown” often commit any of these distortions: (a) disregarding the flaws of the research; (b) misinterpreting it; (c) exaggerating its implications; or (d)cloaking it in vague language. Sometimes they do this without intent of distorting, but the distortions remain.

Let’s take an example that shows all these distortions. Certain teaching methodologies emphasize a combination of gesture, speech, and listening. While such a combination makes sense, it is taken to extremes by Whole Brain Teaching, a rapid call-and-response pedagogical method that employs teacher-class and student-student dialogue in alternation. At the teacher’s command, students turn to their partners and “teach” a concept, speaking about it and making gestures that the partner mimics exactly. Watch the lesson on Aristotle’s “Four Causes,” and you may end up dizzy and bewildered; why would anyone choose to teach Aristotle in such a loud and frenzied manner?

The research page of the Whole Brain Teaching website had little research to offer a few months ago. Now it points to a few sources, including a Scientific American article that, according to the WBT website,  describes “research supporting Whole Brain Teaching’s view that gestures are central to learning.” Here’s an instance of vague language (distortion d). Few would deny that gestures are helpful in teaching and learning. This does not mean that we should embrace compulsory, frenetic gesturing in the classroom, or that research supports it.

What does the Scientific American article say, in fact? There’s too much to take apart here, but this passage caught my eye: “Previous research has shown”—eek, research has shown!— “that students who are asked to gesture while talking about math problems are better at learning how to do them. This is true whether the students are told what gestures to make, or whether the gestures are spontaneous.” This looks like an instance of exaggerating the implications of research (distortion c); let’s take a look.

The word “told” in that passage links to the article “Making Children Gesture Brings Out Implicit Knowledge and Leads to Learning” by Sara C. Broaders, Susan Wagner Cook, Zachary Mitchell, and Susan Goldin-Meadow,  published in the Journal of Experimental Psychology, vol. 136, no. 4 (2007), pp. 539–550. The abstract states that children become better at solving math problems when told to make gestures (relevant to the problems) during the process. Specifically, “children who were unable to solve the math problems often added new and correct problem-solving strategies, expressed only in gesture, to their repertoires.” Apparently, this progress persisted: “when these children were given instruction on the math problems later, they were more likely to succeed on the problems than children told not to gesture.” So, wait a second here. They didn’t have a control group? Let’s look at the article itself.

The experimenters conducted two studies. The first one involved 106 children in late third and early fourth grade, whom the experimenters tested individually. For the baseline set, children were asked to solve six problems of the type 6 + 3 + 7 = ___ + 7, without being given any instructions on gesturing. Children who solved any of the problems correctly were eliminated from the study at the outset. (Doesn’t this bias the study considerably? Shouldn’t this be mentioned in the abstract?)

From there, the students were assigned to groups for the “manipulation phase” of the study. Thirty-three students were told to gesture; 35 were told to keep their hands still; and 38 were told to explain how they solved the problems. The students who were told to gesture added significantly more “strategies” to their manipulation than did the students in the other two groups; however, nearly all of these strategies were expressed in gesture only and not in speech. Across the groups, students added a mean number of 0.34 strategies to their repertoire, 0.25 of which were correct (the strategies, that is, not the solutions).

It is not clear how many students actually gave correct answers to the problems during the manipulation phase. The study does not provide this information.

The second study involved 70 students in late third and early fourth grade; none had participated in the first study. After conducting the baseline experiment (where no students solved the problems correctly), the researchers divided the students into two groups for the manipulation phase. Children in one group were told to gesture; children in the other group were told not to gesture. The researchers chose these two groups because they were “maximally distinct in terms of strategies added.” (How did they know this in advance? This is not clear.)

Again, the students who had been told to gesture added more strategies to their repertoire; those told not to gesture added none.  The researchers state later, in the “discussion” section of the paper: “Note that producing a correct strategy in gesture did not mean that the child solved the problems correctly. In fact, the children who expressed correct problem-solving strategies uniquely in gesture were, at that moment, not solving the problems correctly. But producing a correct strategy in gesture did seem to make the children more receptive to the later math lesson.”

After the children had solved and explained the problems in the manipulation phase, they were given a lesson on mathematical equivalence. (There was no such lesson in the first study.) The experimenter used a consistent gesture (moving a flat palm under the left side of the equation and then under the right side) for each of the problems presented. Then the students were given a post-test.

On the post-test, the students told not to gesture solved a mean of 2.2 problems correctly (out of six); those told to gesture solved a mean of 3.5 correctly. (I am estimating these figures from the bar graph.)

Why would anyone be impressed by the results? For some reason the researchers did not mention actual performance in the first study. In the second, it isn’t surprising that the students told not to gesture would fare worse on the test. A prohibition against gesturing could be highly distracting, as people tend to gesture naturally in one way or another. Again, there was no control group in the second study. Moreover, neither the overall mean performance on the test or the performance difference between the groups is particularly impressive, given that the problems all followed the same pattern and should have been easy for students who grasped the concept, provided they had their basic arithmetic down.

The researchers do not draw adequate attention to the two studies’ caveats or consider how these caveats might influence the conclusion (distortions a and b). In the “discussion” section of the paper, they state with confidence that “Children told to gesture were more likely to learn from instruction than were children told not to gesture.”

This is just one of myriad examples of research not showing what it claims to show or what others claim it shows. I have read research studies that gloss over their own gaps and weaknesses; popular articles that exaggerate the implications of this research; and practitioners who cite the  popular articles in support of their particular method. When I hear the phrase “research has shown,” I immediately suspect that it isn’t so.

*From Shakespeare’s Much Ado About Nothing; thanks to Jamie Lorentzen for reminding me of the phrase.

Wieseltier’s “Going to Melody” and the Language of Lament

We aren’t expected to lament much today. If we have a complaint about the state of things, we’re supposed to back it with data. It will not do, for instance, to say that children don’t have enough time to roam. We must argue, instead, that research has shown that the lack of roaming affects a particular area of children’s brains.

This puts us in a bind. Research has its place, but if it replaces lamentation, we lose something of humanity (and I won’t provide brain data for that). Lamentation has thousands of known years beneath it, and even more unknown years. It has the toughness  of memory and treasure. It is about those things that we will not give up even when the world says we must. It is David weeping for Absalom; it is Jean Passerat’s “J’ay perdu ma tourterelle”; it is the narrator in a Gogol story who exclaims, “It’s dreary in this world, gentlemen!” It is private, but its sorrow and roughness reach us. We thirst for those quirky cries so close to our own.

So it was with thirst that I read Leon Wieseltier’s piece “Going to Melody” (The New Republic, February 2, 2012). It’s about the uninspired greed of large businesses such as Amazon—how their “hunger for profit exploits a hunger for meaning.” As they drive bookstores and record stores to the ground, they kill the activities that take place in those stores, particularly browsing. Browsing is the opposite of searching, writes Wieseltier: “Search is precise, browsing is imprecise. When you search, you find what you were looking for; when you browse, you find what you were not looking for.”

Granted, there is browsing within searching and vice versa. But the distinction stands—and there’s still more too it. Browsing makes room for uncertainty and serendipity, for those books we didn’t expect to find, let alone continue reading. Amazon doesn’t have room for serendipity. “After all,” Wieseltier explains, “serendipity is a poor business model. But serendipity is how the spirit is renewed; and a record store, like a bookstore, is nothing less than an institution of spiritual renewal.”

Where’s his evidence? The memories of Melody Records, for instance, which for thirty years stimulated him and “provided a sanctuary from sadness and sterility.” Or his father’s furniture store, where Wieseltier as a boy sold sofas to U.N. people who lived nearby—by talking to them about the crises in Iran and Cyprus. He knows the foe, too: he describes Amazon’s Price Check, the app that allows customers in a bookstore to scan an item’s bar code and transmit the information to Amazon, which then offers a discount if the store’s price stands to compete with its own.

The evidence lies not only in these memories and details, but in the language of the piece, its amblings and rhythms and visions. As I read it, I too experienced a “sanctuary from sadness and sterility.” I found myself in the record store, listening to the advice of the staff, enjoying this song, or at the bookstore, picking up a book and reading it without rush. I came in with worries and sadness, which loosened and fell away, and with petty complaints, which fizzled in their silliness. I was not loftier than before, but somehow the piece lifted me a little, the way a parent lifts a child up to the window.

Kudos to Wieseltier for not citing brain research here (or any research, for that matter). It would have taken something away from this piece. Yes, there may be research indicating that our loss of real-life interactions has correlated with an increase in stress. Yes, it may be interesting and important. But the point of such a piece is to say, with full risk of disapproval, that I have lost something dear to me, be it a surprising piece of music or a store my friend owned or a place where I can be renewed and revived.

At least we have not yet lost the language for robust lamentation. Let us take care not to lose it. Let us not ask it to be anything else.

Follow

Get every new post delivered to your Inbox.

Join 134 other followers