Research Has Shown … Just What, Exactly? (Reprise)

A few years ago, I wrote and posted a piece with this title, minus the “(Reprise).” (And here’s a piece from 2011.)

It seems apt today (literally today) in light of Dana Carney’s statement, released late last night, that she no longer believes  “power pose” effects are real. She explains her reasons in detail. I learned about this from a comment on Andrew Gelman’s blog; an hour and a half later,  an article by Jesse Singal appeared in Science of Us (New York Magazine).

Dana Carney was one of three authors of the 2010 study, popularized in Amy Cuddy’s TED Talk, that supposedly found that “power poses” can change your hormones and life. (Andy Yap was the third.)

The “power pose” study has been under criticism for some time; a replication failed, and an analysis of both the study and the replication turned up still more problems.  (For history and insight, see Andrew Gelman and Kaiser Fung’s Slate article.) Of the three researchers involved, Carney is the first to acknowledge the problems with the original study and to state that the effects are not real.

Carney not only acknowledges her errors but explains them clearly. The statement is an act of courage and edification. This is how research should work; people should not hold fast to flawed methods and flimsy conclusions but should instead illuminate the flaws.


Update: Jesse Singal wrote an important follow-up article, “There’s an Interesting House-of-Cards Element to the Fall of Power Poses.” He discusses, among other things, the ripple effect (or house-of-cards effect) of flawed studies.


Science ≠ Community

communityI enjoy Andrew Gelman’s blog; it’s a great place if you have heard too many false and flashy “research has shown” and “science tells us” statements and want to know (a) where such research goes wrong and (b) why these errors don’t get attention. The blog attracts readers and commenters from many fields and perspectives; some of the comments could be pieces on their own.

Recently there was an enlightening post followed by lively discussion of Susan Fiske’s call for an end to the reign of “destructo-critics” and “self-employed data police” in social media—that is, those who employ “methodological terrorism” in criticizing others’ research. She doesn’t name names or give any concrete examples, so it isn’t clear who the “destructo-critics” are. She does suggest, though, that the legitimate channels for criticism are peer review or “curated” discussion. That is, she opposes not just nasty tweets, vicious personal attacks, and so forth, but (possibly) any unsanctioned critical commentary on research. In her conclusion, she says, “Ultimately, science is a community,  and we are all in it together.”

A community, eh? That rings a bell….

A commenter (“Plucky”) seized this sentence this and gave it a good shaking:

The key problem with Fiske is this sentence: “Ultimately, Science is a community, and we are all in it together.”

That is just flat-out wrong, and wrong in the ways that result in all that you have criticized. Science is not a community, it is a method.

That’s just the beginning; “Plucky” goes on to explain the dangers of the “community” metaphor. I recommend reading the whole comment. Here’s another choice quote:

My main criticism of this post is the stages of the metaphor—you’re nowhere near six feet of water in Evangeline. If Science devolves into merely a community, then it’s just another political interest group which will be treated as such.

I then remembered the many times I had heard the phrase “the consensus of the scientific community,” along with references to Thomas Kuhn, who supposedly coined it. Kuhn actually said, “What better criterion could there be … than the decision of the scientific group?” He explained what he meant by this, but I consider the statement problematic at best, even in context.

Kuhn aside, I use the word “community” sparingly and cautiously. Many entities that call themselves communities are not communities or should not be. As “Plucky” notes, “communities do not generally value the truth over their members’ well-being.” They exist to support their members.

In fact, someone who wishes to challenge a prevailing idea must often speak independently, without waiting for “community” approval. Dana Carney has just done this in relation to “power poses.” (This is big news, by the way.)

I do not disparage communities overall. Communities of various kinds have a place in the world, and I belong to a few. Still, even the best communities can ask themselves, “To what extent is this a community, and whom do we leave out?” and “What goals does this community not serve, and where does it even counter them?” The point is not to make the community all-encompassing but rather to recognize its limitations.

When it comes to science, I’ll take an open forum over a community any day.


Note: I added the paragraph about Carney after posting this piece.
Update: Writing for
Science of Us (New York Magazine), Jesse Singal reported this morning on Carney’s statement and explanation.

Introversion: Pro-Idea, Anti-Noise, or Something Else?


There has been much discussion of introversion and extraversion but little agreement about what they are. Moreover, I have seen multiple implicit definitions of introversion within the same article or discussion.

It would not matter much, except that some people with power are starting to say, “introverts are this,” “introverts are that,” “introverts need this,” “introverts need that.” Interior designers, engineers, and consultants have been creating “Quiet Spaces” in workplaces. Schools have undergone training to become more introvert-friendly. These initiatives may hold some good but need vigorous (and rigorous) questioning.

In a 2014 article in Scientific American, Scott Barry Kaufman gives a sampling of the many floating definitions of introversion. They run the gamut and then some. He then reveals that psychologists have put forth a model of four types of introversion: social (where you like to be alone or spend time with a few close friends), thinking (where you pay close and continual attention to your own thoughts and feelings), anxious (characterized by self-consciousness and shyness), and restrained (where you tend to think before you act). He then offers a quiz to help you find out which kind you are.

Even there, I see many complications (which he acknowledges as well). To be a “thinking” introvert, must one primarily be interested in one’s own thoughts and feelings, or can one be absorbed in thinking about something else, such as music, a language, or a mathematical problem? The quiz presumes the former, but I object.

As for the other types, when I look at the questions, my response is often, “It depends.” The ambiguity does not bother me; I don’t feel a need to narrow myself down by type and subtype (on other people’s terms). But others are busy doing just that—not for me in particular, but for “introverts” at large.

So, for instance, “Quiet Spaces,” envisioned and designed by Susan Cain and others, exist to give introverts an environment that brings out their best. The intent here is good but the execution narrow. I would not want to work in one, and in this I am not alone. I don’t like the lounge-y feel, the glass walls all around (frosted, but still), the lack of bookshelves, or the colors. Give me a good old office with solid walls, a windowed door, an actual desk, a window to the outside, and plenty of shelves. Or, if space is lacking, just give me a cubicle and some quiet. Again, I see the good intentions but question the assumptions and aesthetic choices.

Nor can a workplace accommodate everyone. I am skeptical of attempts to identify employees’ personality types and tailor workplaces to them. Instead, give the work itself more dimension and beauty. Give people more room to think. Create forums for  discussion. Allow people to work alone, coming together when necessary. Also, let them treat the job as a job, not as an all-consuming career (unless they really want the latter). That way, they can pursue their interests in their own time.

What about schools? Attempts to create introvert-friendly classrooms may also rely on false or skewed assumptions. Some assume that introverts dread speaking to the whole class and prefer speaking to a partner (e.g., in a “think-pair-share” activity); this is not necessarily true, though it may be true for some. There are those who count unequivocally as introverts yet thrive in class discussion, precisely because it is about something interesting. There are those who dread the “think-pair-share” activity because of its “buzz” (so many people talking at once) and its tendency to water down the ideas before they reach the full forum.

Here too, one can reach students by paying attention to the subject matter. When the point of class discussion is to reach greater understanding (about a work of literature, a mathematical concept, or a philosophical idea), students may sit quietly and think, venture a tentative idea, or offer an insight. All of this contributes to the understanding. One lesson might consist primarily of lecture, another of whole-class discussion, and another of a combination (or something different). In each case, students may participate in a variety of ways. Yes, the teacher should be alert to the students but can also trust the subject to lead the way.

And what about the world outside of work and school? Here again, beware of constricting generalizations. I just read an article titled “Introverts Love Facebook, and Extroverts Hate It. Here’s Why.”How does the author justify such a wrongheaded assertion? Here we go:

Everything about Facebook serves the emotional and psychological needs of introverts. It gives them a place to socialize and chat with people they like, without having to deal with the elements of in-person dialogues that make them uncomfortable. It allows them to say their piece, without being interrupted, scowled at, or patronized.

What? Who says introverts are uncomfortable with in-person dialogues? There are those who vastly and vehemently prefer such dialogues to the groupy, chatty, like-y, Facebook-y stuff. I myself dislike Facebook precisely because it’s so social (in Hannah Arendt’s sense of the word). Unless you have a private chat, which tightens you with its tiny windows and bubbles, you have to accept group conversations,  which aren’t even conversations. I recognize the efficiency of Facebook (it helps you stay in touch with many people at once), but it can’t hold a candle to a letter, phone conversation, or conversation in person.

I resist our excessive tilt toward gregariousness, talk, quick answers, busyness, aggressiveness, and so forth. Yet I also resist the push to classify people. Personality research is fine, but those involved should acknowledge its questions and doubts; moreover, they should exercise caution when applying it to policy and products. It is sad to see “groupthink” arising around introversion, when introversion, like extraversion, holds so many variations and possibilities.


Note: I took the above photo at Anne Loftus Playground (around 8 a.m., before children and parents started arriving).

I made a few revisions to this piece after posting it.

Personality Testing on the Job


It isn’t a complete surprise that I missed Eliza Gray’s Time article “How High Is Your XQ?” published on June 22, 2015. I was teaching, it was near graduation time, and my days were filled. But I am surprised not to have seen any outcry over the phenomenon she describes.

According to Gray, employers use personality tests not only to screen potential candidates but to monitor them once they are hired: “Once on board, many of these companies continue to track and crunch data about workers’ personality traits to help find candidates for promotions, transfers, and—at times—termination.”

Here’s what she says two paragraphs later:

Some employers are now monitoring workers’ temperaments in real time—including the world’s largest hedge fund, where employees can track their individual stats on a personal digital “baseball card.” Experts in the fast-growing “people analytics” industry believe it won’t be long before algorithms regularly sift through Facebook and Twitter postings to glean and analyze additional data.”

I have a friend who used to work at Microsoft. One day they had a training session where  they all had to walk around the room wearing a personality color chart. Their co-workers would make marks on their chart to show how they perceived them. Some people found out they had a “blue” personality (according to the majority), others a “red” personality, and so on. Microsoft carried this into their hiring practices; if they thought they had too many “blues” and not enough “reds,” they would give priority to the latter.

This practice is intrusive, flawed, and dangerous. There’s no need to explain the intrusive aspect; who wants to work at a place that is keeping data on your personality from moment to moment? What if you are going through a difficult time—still doing your work well, but dealing with sadness and anxiety throughout the day and trying your best to keep it to yourself? A wise supervisor would leave you alone; the data-gatherers would revise your “baseball card” for everyone to see.

As for the flawed (no, utterly bungled) aspect, this practice applies general statistical patterns to individuals—and, in doing so, affects their employment status and daily lives. Suppose, for instance, that the data indicate that people who have watched the Simpsons are better software coders than those who have not. Does this mean that a person who hasn’t watched the Simpsons will necessarily be a lesser coder? Of course not. But people do get weeded out for silly, arbitrary, obscure things like that.

Why is this dangerous? In giving so much credence to the general drift of data, these practices disfavor the outlier, the one who contradicts the trends. Moreover, companies generally keep their algorithms secret; an employee or candidate cannot find out exactly what is being judged and how.

I have written about this general issue before, in my book and on this blog, but did not realize it had gone to such extremes. This needs not only protest and pushback, but some good lawyers.


Image credit: Color Code, “Personality Test.”

Time, Money, Happiness, and a Bit of Skepticism

A New York Times article by Hal E. Hershfield and Cassie Mogilner Holmes describes a study of the relationship between one’s desire for more time or money and one’s happiness. The authors, who conducted the study, conclude that those who value time over money tend to be happier than those who value money over time. The authors even called the results “predictive”:  “But our research does show that the value individuals place on these resources relative to each other is predictive of happiness.”

Now, I would love to believe this. It makes sense to me. Maybe those who value time over money are happier with their lives to begin with  (why else would they want time?). Or else, focusing on the things they want to do (rather than the things they want to buy), they feel relatively happy. But the study teems with problems. I downloaded the full paper; it is long and convoluted, so I did not read all of it. But even a cursory glance left me skeptical. It isn’t something I’d lap up.

First, I question the use of the word “predictive.” It suggests a much stronger correlation than actually came through in the study. Also, it suggests a relation between present and future: At least some lay readers (including myself) would take “predictive” to mean “indicative of a future state.”The meaning of “predictive” should  be clarified.

Beyond that, the study has some serious murk. (For those who want to take a look, it is published in the September 2016 issue of Social Psychological and Personality Science.)

The researchers considered the possibility that those who value money over time (that is, who say they would rather have more money than more time) are actually under financial stress and thus less happy than those who choose time. To control for this possibility, they asked participants to rate their time and income both subjectively (in terms of how they felt about it) and objectively (in terms of hours worked per week and salary).

Now, one’s work commitments may be only a fraction of one’s overall commitments–so “hours worked per week” give only a weak indication of one’s free and occupied time. Similarly, salary by itself does not say much about one’s financial resources. Cost of living, spending habits, financial responsibility toward others, etc. also play a large role. So I am not sure that this control was effective.

Then, at the end of the study, the researchers took a surprising turn. To test for manipulation, they conducted two new and separate surveys with new participants. In the second one (4b), they assigned participants randomly to three groups. One group was told to write about the ways in which they were happy and satisfied with their lives. The second was told to write about ways in which they were unhappy. The third (control) group did not have a writing task. They were then all asked which they wanted more of, time or money, and were asked to rate their  happiness/satisfaction, report on their income and time, etc.

It turned out that those in the “happy” condition (Group 1) chose time over money and rated themselves as happier than those in the “unhappy” condition, who chose money over time. There were no differences in choice between those in the “unhappy” and control conditions. The authors comment (in the study itself): “These results are suggestive of a causal relationship whereby being happier makes one more likely to prefer time over money than being unhappy.” This, along with the results of another survey, pointed to a bidirectional effect. Yes, maybe, but it also suggests that happiness ratings are not only subjective but flimsy, easily influenced by the immediate circumstances.

It would be more telling, maybe, if people rated their happiness over a period of several years. But that would be much more difficult to accomplish. (Study 3b, which used Qualtrics rather than Mechanical Turk, did involve a follow-up one year later, but it apparently asked only about time vs. money, not about happiness, and the consistency with the previous 3b was only moderate.)

For five of the seven surveys in this study, the participants were recruited through Amazon’s Mechanical Turk. That in itself does not inspire confidence. (I wonder, moreover, whether those who complete Mechanical Turk surveys for the money might be in a different demographic from those who do it for fun. That would influence the results.)

Few people outside of psychological research circles are going to read the whole study. It is not available online except through subscription to the journal (or an institution’s subscription), and it’s a dense read. But it doesn’t take advanced knowledge to spot some big flaws. In fact, the study’s multiple comparisons, hierarchical regressions, etc. do not succeed in masking its fundamental silliness. It reminds me, in fact, of the brilliant satire “NO TRUMP!: A Statistical Exercise in Priming” by Jonathan Falk and Andrew Gelman.

Why is the study fundamentally silly? Because of things like this (the description of the one survey in the study that was conducted in person):

Participants (N= 535) were recruited in the train station of a major East Coast city to complete a brief survey in exchange for a granola bar. One hundred and six people who did not complete the survey were excluded from the analyses, leaving a final sample of 429.

Come now. If, of those who agree to take the survey, nearly 20 percent do not complete it, how seriously can they be taking it? (I  realize that people drop out of studies frequently, but given that this survey was brief and administered in person, the completion rate seems a bit low.) Moreover, the researchers found that the train station participants were generally happier (and valued time more) than the participants in the Mechanical Turk surveys; doesn’t that suggest something wobbly in the methodology? (Also, their average income was considerably higher than that of the other participants.)

Again, the idea has merit.  I wish the study lived up to it. There’s much more to say about the problems, but I will stop here, since I value my time.

Addendum (since I kept on thinking about it anyway): It appears that in each of the surveys, participants were asked about their happiness after they answered the other questions. A good replication test would involve the reverse: Have them answer questions about their happiness first.

Note: I made a few minor revisions after posting this piece.

TEDx Video Coming Soon

The video of my talk at TEDx Upper West Side is now complete and will be uploaded soon. The title, “Take Away the Takeaway,” is the working title of my forthcoming book.


“Go to Peoria”

Yesterday I came upon the story “The Ghostwriter” by William Lychack, whose work I had never read before. It starts with a man who has heard the voice of God saying, “Go to Peoria”—and who has followed that voice. Within the space of five pages, the narrator, a ghostwriter, takes those three words and changes them into something I didn’t expect, not the opposite of my expectation, but something more like “that and its opposite and something else.” When a writer can do this, it’s no fluke—so I intend to read his collection The Architect of Flowers. I am holding off just a little until I have made more headway through my current readings and kept to my budget for the month.

The story got me thinking about how stories in general work. I love the distillation of the short story, the way it makes the most of its time. A good story is a riddle of sorts; it bares itself in a surprising way. Also, this story plays intriguingly with the idea of a takeaway; the ghostwriter’s work is all about takeaways, but in his work he goes beyond his work.

We often think of “going beyond” one’s work in external terms (working more hours, taking on more tasks, etc.), but one can go beyond by going inward, into the subject and principles of the work. This is the neglected part of teaching: thinking about the subject matter and the lessons. Not attending meeting after meeting, but thinking and reading. A performer “goes beyond” by practicing and practicing until the fluency itself opens up the piece in new ways. He reaches a new place of entry. It is not mystical in practice; it comes from persistent work. But the work is not “busy”; in fact, it leaves the busyness behind.

“A Time When I Can Think Slowly Through Things”

Last spring I went to the New York Philharmonic to hear Schumann’s Cello Concerto. Carter Brey was the soloist; his rendition thrilled me with its subtlety and dialogue. (For years, Rostropovich’s interpretation was by far my favorite; Brey’s went beyond it.) I went back a second time, for the final night, and was sorry I couldn’t go back again.

So I was delighted to find a video clip of the New York Philharmonic rehearsing the concerto in Costa Mesa. The clip is much too short (just a fraction of the second movement), and I wish that the video editor had shown more of the musicians instead of including those city views. Even so, it’s great to watch and hear. The duet with Eileen Moon is gorgeous, and those few seconds of rehearsal accomplish and convey a lot.

While on this search, I found two excellent interviews: one with Noah Rothbaum in Runner’s World and the other with Tim Janolt for the Internet Cello Society. There are many more, but I had to limit myself. These two are full of interesting things. Brey describes running as “a time when I can think slowly through things.” He says of Laurence Lesser, his first cello teacher in college, that “his most valuable gift was showing me how to think for myself in order to find solutions to technical problems in a non-dogmatic manner.”

Here’s a quote from the first interview:

Is Bach better to listen to before running or Beethoven?
For a classical musician, great classical master works don’t really work as background music. We all find that when restaurants put classical pieces that we know on as soft background music, it’s a tremendous annoyance to us because we just want to stop and listen. The volume is usually just below the threshold for you to hear clearly. We find it annoying and offensive because this is music that wasn’t meant for background music. So it depends on what you need. If you’re really in the mood to concentrate on something that’s complex, that has certain surface complexity, then I’ll put on a piece of classical concert music. If I need something mindless to get my spinal cord going then I’ll put on pop music.

Hear, hear! And from the second:

TJ: How does one shift “in character” with the music?

CB: When shifting between two notes, many cellists tend to be on the late and fast side, which may serve musical purposes at times, though it often doesn’t. This kind of shifting is more utilitarian, merely getting from point A to point B, since it is but one of an infinite number of ways of going between two notes. It’s better if you can more consciously decide how much of a slide you want to hear. If you want to hear more of a broad-reaching kind of slide, don’t shift so late; leave the first note earlier so that there’s a more vocal effect in getting to the goal note. For wonderful examples of this, listen to the great singers, like Jessye Norman and Dietrich Fischer-Dieskau, who were also great influences on my development.

I look forward to reading more, but much more than that, to hearing more.

On Beginnerhood


Yesterday I went kayaking again and managed to take a photo from the unusually tippy boat. The first time I went, I was charging ahead with confidence; this time, I wobbled and veered. I can blame the boat, but the truth is that I don’t have technique yet. The first boat was more forgiving. (Two very kind volunteers gave me  a little lesson; by the end, I was making good progress.)

Having been a beginner at many things, from languages to electronics, I can speak to some of its joys:

In a short time you can move from knowing nothing to knowing something (and seeing that there’s still much more to learn). That can be exhilarating.

You can usually do something with what little you know. That includes thinking about it. This means the mind has more good things to carry around.

Initially, there’s a certain charm in ineptitude, and others treat it generously.

Then come the drawbacks:

The charm of ineptitude fades quickly; after that, there’s nothing but excellence to strive for, and little chance of reaching it.

Beginners struggle to perform even simple tasks, like rowing, saying a sentence in a new language, or playing a simple melody. More work for less beauty doesn’t seem fair.

For the most part, beginners know that they can progress if they practice long and well. It may take considerable time. Perpetual beginners have chosen, in some way and for some reason,  not to take on that commitment. This can be embarrassing to admit.

All that said, it’s good that there’s room for beginners, even perpetual beginners, in the world. There’s only so much that we can do well, and it would be a shame to give up the rest. I may never be an expert kayaker, but I hope to go out on the water many more times in my life.

Bad Lemon Logic

Back in the 1960s, Hans Eysenck and Sybil Eysenck conducted an experiment that suggested that introverts (identified through a questionnaire) salivated more than extraverts when exposed to lemon juice, presumably because they have a higher baseline level of cortical arousal. The results were widely popularized; in an interview with Scientific American, Susan Cain said, “Introverts even salivate more than extroverts do if you place a drop of lemon juice on their tongues!”If you Google “lemon introvert salivation” you will see thousands of mentions of the study and minimal critical discussion.

Now, this study has problems; later studies called its findings into question. (What nerve! I think of Andrew Gelman’s “Enough with the Replication Police.”) I intend to dust off my own police uniform and look into all of this. For now, I will focus on the error that comes up again and again in interpretations of this test. People now claim that you can find out how introverted you are by conducting the lemon experiment. That is not only preposterous but illogical.

It is one thing to say that a study suggests that introverts tend to salivate more than extraverts in response to lemon juice. I question such a claim and the rigor of the study that led to it, but that’s what you’re supposed to do with such studies anyway. Now, to claim the reverse—that you can find out how much of an introvert you are by putting lemon on your tongue and measuring your saliva output—is to succumb to the famous fallacy of affirming the consequent.

Here’s why it’s wrong. Studies like the lemon juice experiment draw general conclusions from an array of individual results. Within the experiment, there may have been introverts who salivated less than extraverts. There may have been quite a few introverts and extraverts who salivated at similar levels. It might even be the case that if you divided salivation levels into two groups, a “low salivation group” and a “high salivation group,” you would find comparable numbers of introverts and extraverts in each. In no way does the test even suggest that if you salivate a lot, then you are an introvert.

Who is claiming such a thing, anyway? The BBC declares, “The amount of saliva you produce after putting a drop of lemon juice on your tongue might tell you something about your personality.” (Shame on them!) But that article has no listed author; it’s possible an intern wrote it. I give the BBC the benefit of the doubt it failed to cast on itself.

I see no excuse, though, for the famous TED-talking scholar Brian Little, who writes in Me, Myself, and Us: The Science of Personality and the Art of Well-Being (2014):

One of the more interesting ways of informally assessing extraversion at the biogenic level is to do the lemon-drop test. [Description of experiment omitted from present quote—DS.] For some people the swab will remain horizontal. For others it will dip on the lemon juice end. Can you guess which? For the extraverts, the swab stays relatively horizontal, but for introverts it dips. … I have done this exercise on myself a number of times, and each time my swab dips deeply. I am, at least by this measure, a biogenic introvert.

Someone of Little’s stature and renown should exercise more responsibility. He not only generalizes “introverts” and “extraverts” but suggests that you can conduct this experiment on yourself and find out who you are. The media (with exceptions) drools over this sort of thing; perhaps bad reasoning is a lemon, and perhaps the press as a whole has high cortical arousal.