The Gift of Criticism

norman-rockwell1A few years ago I edited a student’s piece on Machiavelli; I had recruited it at the last minute for my students’ philosophy journal CONTRARIWISE and found that it needed clearer wording in places. When I presented him with the edits, he said that he accepted some but not all of them: that in a few places he was trying to say something else. We sat down to discuss this. In telling me what he meant, he found the right wording; by the end of our meeting, he had revised the piece to his satisfaction. This happened because he was open to the suggestions but strong enough to make his own decisions. Also, I saw past the particulars of my edits; I wanted to help him find his words, not replace them with mine.

This memory returns as I ponder two recent articles about Amy Cuddy and the power pose: Susan Dominus’s New York Times piece and Daniel Engber’s response in Slate. I find Engber’s article much clearer and more to the point–but he also has the benefit of hindsight, critique, and revision. Dominus may well follow up with some afterthoughts. She tackled a complex and heated topic and (from what I can see) did her best to present it fairly. Yet the article fails to distinguish adequately between personal attack and criticism. I posted a comment, which I am developing a little further here. This piece is not about Cuddy; it’s about criticism itself. (Regarding the power pose study, there are numerous recent comments–from many perspectives–in the article’s comment section and on Andrew Gelman’s blog.)

Here’s the key difference, as I see it, between criticism and personal attack: criticism gives you something concrete to consider, something about the issue at hand, be it your work, your actions, or even your personality. Its aim is to point out areas for improvement. It is not always correct or kind; sometimes critics can be vehement and unsympathetic, and sometimes they make mistakes or show biases. But if it is about the thing itself, if it analyzes strengths and weaknesses in a coherent way, it counts as criticism. By its nature it points toward improvement. It is not necessarily negative; it can recognize strengths and excellence.

Personal attack does not give you a chance to improve. Maybe it comes in the form of vague and veiled hints. Maybe it’s incoherent. Maybe it focuses on your personal life instead of the issue at hand. Maybe it gets said behind your back, without your knowledge. Or maybe it’s about something so fundamental to you that it’s unfair to expect you to change. In any case, when it comes to helpful content, there is no “there” there, at least no “there” that invites you in.

In that light, criticism is a gift, even when the delivery is not ideal. It offers working material. But our culture is not well attuned to criticism; we’re taught to hear the “yay” or “nay,” the “up” or “down,” not the subtler responses. For criticism to achieve its purpose, several conditions must exist.

First, institutions would have to make generous room for error, reexamination, and correction. Universities, schools, scientific organizations, publications should not only acknowledge error openly but treat it as part of intellectual life, not cause for shame or demotion.

Second, the person giving the criticism should do so as frankly and humbly as possible: laying the critique on the table without claiming superiority. There’s some disagreement over whether this should happen in private or public, by in person or online. As I see it, a published work can be criticized anywhere–online or offline, in public or private–but an unpublished work or private act should receive more discreet treatment. Published books get reviewed publicly, after all; there’s no suggestion that a reviewer should contact the author privately before saying something in the New York Review of Books. But if I send someone an unpublished manuscript for comment, I expect this person to reply to me alone (or me and my editor) and not to the world.

Third, the person receiving the criticism should learn to hear it and separate it from the emotion it may stir up. Even thoughtful, carefully worded criticism can be hard to hear. It takes some strength to sort out the upset feelings from the actual content of the words. It takes even more to decide which parts of the criticism to take, which to reject, and which to continue considering. Some criticism incites us to reconsider everything we have done; some draws attention to small (but important) details. To hear and use criticism well is to open oneself to profound improvement.

Just before the final manuscript of Republic of Noise was due, someone who read the manuscript offered me some far-ranging suggestions. I saw her points but didn’t want to apply them rashly, in a rush. To decide whether, how, and where to apply them, I would need much more time than I had. I decided to keep them in mind for the future. I am glad of this decision; the book was the way I wanted it, but her suggestions helped me with subsequent writing.

Why do I say that our culture isn’t set up well for criticism? We aren’t taught how to handle it. As a beginning teacher, I remember being told (at numerous “professional development” sessions) not to use red pen, since it could make a student feel bad; not to write on students’ work, but to use Post-its instead; and to limit the comments to two commendations and two general suggestions for improvement. While some of the gist is good (one should avoid overwhelming students or treat one’s own appraisal as the last word), it assumes students’ extreme fragility in the face of concrete, detailed suggestions. The more we treat criticism as devastating, the more fragile we make ourselves (both the critics and the recipients).

Hearing criticism–actually perceiving and considering its meaning–deserves continual practice. It requires immersion in the subject itself; you can’t practice criticism without practicing the thing criticized. It isn’t always fun, but it can lead to exhilaration: you see, on your own terms, a way of doing things better.

 

Image: Norman Rockwell, Jo and Her Publishor (this title may or may not be correct; I have also seen it as Jo and Her Publisher and Jo and Her Editor). This is one of his several illustrations of Louisa May Alcott’s Little Women. In Chapter 14, Jo publishes two of her stories in a newspaper.

I made a few revisions and additions to this piece after posting it.

A Lesson from the Power Pose Debacle

Amy Cuddy’s TED talk on power posing has thirty-seven million views. Its main idea is simple: if you adopt an expansive, authoritative pose, your actual power will increase. For evidence, Cuddy refers to a study she conducted in 2010 with Dana Carney and Andy Yap. Holes and flaws in the study have since been revealed, but Cuddy continues to defend it. Doubt fuels scientific inquiry, but in an era TED-style glamor and two-minute “life hacks” (Cuddy’s own term for the power pose), we find a shortage of such doubt on stage. It is time to tap the reserves.

Recently TED and Cuddy appended a note to the summary of the talk: “Some of the findings presented in this talk have been referenced in an ongoing debate among social scientists about robustness and reproducibility.” In other (and clearer) words: The power pose study has not held up under scrutiny. At least two replications failed; Andrew Gelman, Uri Simonsohn, and others have critiqued it robustly; and Carney, the lead researcher, detailed the study’s flaws—and disavowed all belief in the effect of power poses—in a statement posted on her website. Jesse Singal (New York Magazine) and Tom Bartlett (The Chronicle of Higher Education) have weighed in with analyses of the controversy.

Very well, one might shrug aloud, but what should we, irregular members of the regular public, do? Should we distrust every TED talk? Or should we wait until the experts weigh in? Neither approach is satisfactory. When faced with fantastic scientific claims, one can wield good skepticism and follow one’s doubts and questions.

Before learning of any of this uproar, I found Cuddy’s talk unstable. Instead of making a coherent argument, it bounces between informal observations, personal experiences, and scientific references. In addition, it seems to make an error early on. Two minutes into her talk, Cuddy states that “Nalini Ambady, a researcher at Tufts University, shows that when people watch 30-second soundless clips of real physician-patient interactions, their judgments of the physician’s niceness predict whether or not that physician will be sued.” Which study is this? I have perused the Ambady Lab website, conducted searches, and consulted bibliographies—and I see no sign that the study exists. (If I find that the study does exist, I will post a correction here. Ambady died in 2013, so I cannot ask her directly. I have written to the lab but do not know whether anyone is checking the email.)

In separate studies, Ambady studied surgeons’ tone of voice (by analyzing subjects’ ratings of sound clips where the actual words were muffled) and teachers’ body language (by analyzing subjects’ ratings of soundless video clips). As far as I know, she did not conduct a study with soundless videos of physician-patient interactions. Even her overview articles do not mention such research. Nor did her study of surgeons’ tone of voice make inferences about the likelihood of future lawsuits. It only related tone of voice to existing lawsuit histories.

Anyone can make a mistake. On the TED stage, delivering your talk from memory before an enormous audience, you have a million opportunities to be fallible. This is understandable and forgivable. It is possible that Cuddy conflated the study of physicians’ tone of voice with the study of teachers’ body language. Why make a fuss over this? Well, if a newspaper article were to make such an error, and were anyone to point it out, the editors would subsequently issue a correction. No correction appears on the TED website. Moreover, many people have quoted Cuddy’s own mention of that study without looking into it. It has been taken as fact.

Why did I sense that something was off? First, I doubted that subjects’ responses to a surgeon’s body language predicted whether the doctor would be sued in the future. A lawsuit takes money, time, and energy; I would not sue even the gruffest surgeon unless I had good reason. In other words, the doctor’s personality would only have a secondary or tertiary influence on my decision to sue. On the other hand, it is plausible that doctors with existing lawsuit histories might appear less personable than others—if only because it’s stressful to be sued. Insofar as existing lawsuit histories predict future lawsuits, there might be a weak relation between a physician’s body language and his or her likelihood of being sued in the future. I suspect, though, that the data would be noisy (in a soundless kind of way).

Second, I doubted that there was any study involving videos of physician-patient interactions. Logistical and legal difficulties would stand in the way. With sound recordings—especially where the words are muffled—you can preserve anonymity and privacy; with videos you cannot. As it turns out, I was flat-out wrong; video recording of the doctor’s office has become commonplace, not only for research but for doctors’ own self-assessment.

It matters whether or not this study exists—not only because it has been taken as fact, but because it influences public gullibility. If you believe that a doctor’s body language actually predicts future lawsuits, then you might also believe that power pose effects are real. You might believe that “the vast majority of teachers reports believing that the ideal student is an extrovert as opposed to an introvert” (Susan Cain) or that “the whole purpose of public education throughout the world is to produce university professors” (Ken Robinson). The whole point of a TED talk is to put forth a big idea; alas, an idea’s size has little to do with its quality.

What to do? Questioning Cuddy’s statement, and statements like it, takes no special expertise, only willingness to follow a doubt. If TED were to open itself to doubt, uncertainty, and error—posting corrections, acknowledging errors, and inviting discussion—it could become a genuine intellectual forum. To help bring this about, people must do more than assume a doubting stance. Poses are just poses. Insight requires motion—from questions to investigations to hypotheses to more questions.  This is what makes science interesting and strong.  Science, with all its branches and disciplines, offers not a two-minute “life hack,” but rather the hike of a lifetime. With a mind full of doubt and vigor, one can make it.

Update: TED has changed the title of Cuddy’s talk from “Your Body Language Shapes Who You Are” to “Your Body Language May Shape How You Are.” In addition, the talk’s page on the TED website has a “Criticism & updates” section, last updated in August 2017. Both are steps in the right direction.

Note: I originally had the phrase “two-minute life hack” in quotes, but Cuddy’s actual phrase is “free no-tech life hack.” She goes on to say that it takes requires changing your posture for two minutes. So I removed “two-minute” from the quotes.

What Does “Predict” Mean in Research?

In her TED talk, Amy Cuddy says,

Nalini Ambady, a researcher at Tufts University, shows that when people watch 30-second soundless clips of real physician-patient interactions, their judgments of the physician’s niceness predict whether or not that physician will be sued.

This is quoted all over the place, yet I have been unable to track down the study. Maybe it is unpublished or in press, or maybe it is published under a title that doesn’t mention physicians or videos.

In the meantime, I wonder whether Cuddy might have conflated two separate studies by Ambady: one of surgeons’ tone of voice (2002), and another of soundless clips of teachers (1993).

But my greater concern is with the word “predict.” As Cuddy puts it, the “judgments of a physicians’ niceness” actually predict whether or not that same physician will be sued in the future. To determine this, a researcher would have to follow the physicians over the long term and compare their subsequent lawsuit patterns (if any) to the initial ratings.

That would be terrifically difficult to accomplish. First, doctors with malpractice litigation histories are a small percentage of the whole, so the sample size would be tiny. Second, how long do you wait for a doctor to be sued? Two years? Five? Ten? Indefinitely?

Instead, I suspect the study followed a procedure similar to that of “Surgeons’ Tone of Voice.” (If I find out I am wrong, I will post a correction.) That is, the ratings of the videos were related to doctors’ existing lawsuit history. If there was any prediction, it was retrospective; as the authors state in the surgeon study, “Controlling for content, ratings of higher dominance and lower concern/anxiety in their voice tones significantly identified surgeons with previous claims compared with those who had no claims” (emphasis mine).

Why does this matter? There’s a big difference between predicting past and future events. People are easily dazzled by the idea that a thirty-second video clip (or sound clip, or whatever it may be) can predict a future lawsuit. The idea that it might predict a person’s existing history is perhaps interesting (if it holds up under scrutiny) but less dazzling.

When it comes to doctors, I imagine those with a lawsuit history might be a little grumpier than the others. I can also see how tone of voice could affect patients’ sense of trust and comfort. But I know of no study that demonstrates that ratings of soundless video clips of physicians predict whether they will one day be sued.

To avoid turning scientific research into a magic show, use the word “predict” carefully and precisely. Also, give a little more detail when referring to research, so that those interested can look up the study in question.

Update: I just learned that Ambady died in 2013. See the comments below. Thanks to Shravan Vasishth for the information.

Another update: Thanks to Martha Smith for explaining various  kinds of “prediction” in statistics. (See here and here.)

Research Has Shown … Just What, Exactly? (Reprise)

A few years ago, I wrote a piece with this title, minus the “(Reprise).” (And here’s a piece from 2011.)

It seems apt today (literally today) in light of Dana Carney’s statement, released late last night, that she no longer believes  “power pose” effects are real. She explains her reasons in detail. I learned about this from a comment on Andrew Gelman’s blog; an hour and a half later,  an article by Jesse Singal appeared in Science of Us (New York Magazine).

Dana Carney was one of three authors of the 2010 study, popularized in Amy Cuddy’s TED Talk, that supposedly found that “power poses” can change your hormones and life. (Andy Yap was the third.)

The “power pose” study has been under criticism for some time; a replication failed, and an analysis of both the study and the replication turned up still more problems.  (For history and insight, see Andrew Gelman and Kaiser Fung’s Slate article.) Of the three researchers involved, Carney is the first to acknowledge the problems with the original study and to state that the effects are not real.

Carney not only acknowledges her errors but explains them clearly. The statement is an act of courage and edification. This is how research should work; people should not hold fast to flawed methods and flimsy conclusions but should instead illuminate the flaws.

 

Update: Jesse Singal wrote an important follow-up article, “There’s an Interesting House-of-Cards Element to the Fall of Power Poses.” He discusses, among other things, the ripple effect (or house-of-cards effect) of flawed studies.

 

The Cult of the Spiffy Solution

Just as the world is made up of two kinds of people–those who divide the world into two kinds of people and those who do not–so it is divided into those who want spiffy solutions to complex human problems and those who do not.

Why, in our era of self-help, TED, and aggressive innovation, would anyone turn down a quick solution to a persistent human problem? Why would anyone cringe at the assertion, “One tiny change can revolutionize your life”? Well, I do.

Perhaps I suspect that the assertion is false; perhaps I wouldn’t even want it to be true. The first possibility is a little easier to explain, so I will tackle it first.

People who propose some seemingly simple solution are often trying to sell it. That is, their solution comes with a book, a program, a product. They not only divulge the solution but take on the role of master coach. This tends to interfere with their ability to criticize their own solution. They have a great stake in promoting it, and they get attention, praise, and money for doing so.

A case in point (one of many examples): Amy Cuddy, a professor and researcher at Harvard Business School, claims that we can change our body chemistry and our behaviors just by changing our posture (that is, by adopting “power poses“). She cites her own research and the research of others in support of this theory.

Her TED talk is listed as one of the most popular of all time. It is already media, but the media lapped it up, in a typical gesture of self-potation. In this talk, she claims that a change of posture–adopted right now–can change the trajectory of your life. She begins: “So I want to start by offering you a free no-tech life hack, and all it requires of you is this: that you change your posture for two minutes.” She ends:

So I want to ask you first, you know, both to try power posing, and also I want to ask you to share the science, because this is simple. I don’t have ego involved in this. Give it away. Share it with people, because the people who can use it the most are the ones with no resources and no technology and no status and no power. Give it to them because they can do it in private. They need their bodies, privacy and two minutes, and it can significantly change the outcomes of their life.

Share the science! Yet when Eva Ranehill and others attempted to replicate her research, they found that power poses had no effect on hormonal levels or any of the behavioral tasks. (They did effect people’s self-reported feelings of power.)

In a Slate article, Andrew Gelman and Kaiser Fung criticize the overall lack of suspicion around Cuddy’s theory and others like it. While not blaming Cuddy herself (or any particular reporter or media outlet), they warn against fuzzy acceptance of so-called “science” (which may be no more than a popular theory that feels good and has some basis in anecdote). Gelman (of whom I am becoming a cautious and questioning fan) is even funnier and more caustic in his own blog post on the subject.

There is every reason to doubt the “power pose” theory and others of its ilk. It’s good to have good posture; this is no innovation. One can strive for good posture, and enjoy its benefits, without pretending that it will catapult every one of us to power.

Now I come to the second point: I wouldn’t even want this thing to be true. Granted, what I want to be true shouldn’t affect what I think is true. Yet it often does; I must always be vigilant about the spill of preference into perception. The question then is: What accounts for the difference between those who want the “power pose” theory to be true and those who don’t?

I don’t want it to be true because I don’t want to be reduced to a success cartoon. Suppose power posing did change hormonal levels and increase risk-taking. Suppose those changes led to power positions, power lunches, power dates. All that power would get boring, and the substance would seep away. I would go on a long search for someone not entirely self-assured and an occupation that did not require constant flashes of confidence.

I don’t mean that others want to become auto-CEOs; I don’t fully understand the allure of the quick “scientific” fix. It seems to skip over two challenges: understanding science itself (at least enough to evaluate the research) and dealing with the complexities of life. Maybe there is ease in the skipping. Ease itself isn’t bad; most of us covet ease of some kind. This is one kind, though, that I don’t covet.

I question the notion of power as ceaseless good. Even in a job interview, some ebb and flow of power is probably ideal; one doesn’t want to tower over the interviewer. Moreover, an interview should be about the job and the candidate’s qualifications. If we are tilting toward a culture of “power impressions,” maybe it’s time to take a few decades, or even a century, to correct that tilt. That one century could change a life.

 

(I made some revisions to this piece after posting it. For some uproarious evening reading, see “NO TRUMP!: A Statistical Exercise in Priming,” which Gelman co-wrote with Jonathan Falk.)