Where Personality Quizzes Go Wrong

Sometimes I take an online personality quiz for fun. It isn’t a good idea. Even flattering results leave me discouraged; I question the quiz’s authority but can’t talk back to it.

Of course some quizzes are meant for amusement, but others pretend to reveal some truth about your nature–even in ten or fewer questions. They even claim a scientific basis.

Here’s where such quizzes miss the mark. First, their claims far exceed their capabilities; they are crude instruments, yet their authors suggest that they can tell you something useful. There’s an inherent discrepancy between the pretension and the actual capability of such a quiz.

Second, the assumed categories (such as personality types) have inherent limitations. Such quizzes, by their very contents, tell only part of the truth. You might find out that you are “an” introvert when in fact you have a mixture of introverted and extraverted tendencies. (Now that Jonathan Cheek and colleagues are positing four kinds of introverts, there are quizzes to tell you which kind you are, as though you had to be one.)

Third, many such quizzes ask you what is “usually” true, when in fact your exceptions may play a large role in your life. For instance, I score low on neuroticism tests, because I do not worry much on the whole. When I do worry about something, I though, I can worry intensely. A quiz’s emphasis on “usual” or “average” behavior doesn’t capture this.

Fourth, the multiple-choice options may not apply to you at all. Sometimes they are directed at a different demographic from yours. Sometimes the authors didn’t consider all the possibilities. I find myself choosing the “least wrong” option instead of one that really suits me.

Fifth, many responses are contextual. Whether or not I enjoy a party really depends on the party. Whether I feel energized by others’ company depends largely on who they are and what else is going on. Sometimes it isn’t possible to give a true general response, yet the quiz requires it.

Sixth, the quiz may rest on shaky theoretical principles. For example, the MBTI was once standard fare but has come under heavy criticism. I will not be surprised if other tests come to a similar end.

Seventh, a questionnaire of this sort has value only when the results are analyzed intelligently. Doctors give questionnaires to help them with preliminary diagnoses; they treat the responses as a starting point, not the last word. No questionnaire should stand on its own as an arbiter of human nature.

Eighth, none of this comes close to the descriptions and characterizations found in literature. No personality test can approximate Pushkin’s poet, Melville’s Ahab, Gogol’s Chichikov, or Bellow’s Tamkin. The difference? In literature, you meet characters who may remind you of people you know, including yourself, but at the same time resemble no one. They let you do the same; when reading, you can find both affinity and liberty; you wake up into imagination.

Psychology has much to offer, but its quizzes tend toward hubris. Annie Murphy Paul says it well in The Cult of Personality Testing: “Personality tests take wildly different forms–questionnaires, inkblots, stories, drawings, dolls–but all make the same promise: to reduce our complicated, contradictory, changeable selves to a tidy label. These tests claim to measure not what we know, but what we’re like; not what we can do, but who we are.”

Even if they seem just silly, they make their way into workplaces, schools, and language. They play to the desire for “interactive” science and quick results; it seems cool to go online, take a test, and find something out about yourself in five minutes. They are basically the candy of self-knowledge, candy that too often gets served up as the main course.

Self-knowledge is much more than “I am this” and “I am that.” It requires long study of things outside the self; it requires participation in the world. A series of clicks won’t get you there.

Can You Prove a Theory True?

The question is partly rhetorical. You’re supposed to get affronted and say, “No, of course you can’t. Everyone should know that. You can only prove a theory false.”

But to dig into the question properly, you have to define “theory” and consider the difference between closed and open systems. The word “theory” is used in many different senses; as Marcelo Gleiser points out, you must observe its context to understand its meaning.

A theory is more than a hypothesis, which in turn is more than a guess or one-time prediction. A hypothesis is a tentative explanation of a general phenomenon: for instance, “Rain occurs more frequently where land is cultivated.” This can later be developed into a theory (e.g., “Rain follows the plow“). Sometimes there is disagreement over whether something should be called a hypothesis or a theory–but the distinction remains.

A theory must be well substantiated, have broad application, and explain a general phenomenon or class of phenomena. When a theory takes the form of an apparently unwavering principle, it may be called a law.

If I say, “There is oil on Mars,” I am not positing a theory; I am just making an untested assertion. If I were to say instead, “Where rocks on a planet show chemical composition X, there is oil beneath the planet’s crust,” that would be a hypothesis; with strong basis and explanation, it could be a theory. If I were to find a unifying principle predicting the presence or absence of oil, I might call it a law. Any of these can be refuted: assertion, hypothesis, theory, and law. Only the assertion can be proven true, in the case that oil is found on Mars. Even then, there are caveats.

Now, growing up in a family of mathematicians, I assumed in childhood that you could prove a “theory” true (provided that your axioms were true). Because mathematics works within a closed system, you can work logically from axioms to conclusions and thereby demonstrate that the latter proceed from the former. (In mathematics, it is a “theorem” that you prove, not a “theory.” The word “theory” in mathematics usually refers to a body of knowledge. The usage is not entirely consistent, though; one hears of Ramanujan’s theory of primes, for instance.)

In the natural sciences, you never have a completely closed system, except in the theoretical fields. This is where things get tricky. In theoretical physics, for instance, you can determine from your axioms and laws how  your model will behave. Models do not completely match the natural world, though. In the natural world, there are mitigating factors (and the truth of the axioms matters a great deal).

The social sciences may be the farthest from any closed system–because so many factors, past and present, can influence human behavior. You usually work with high degrees of uncertainty. What do you do? Do you just give up? There are those who believe the social sciences are pure nonsense, but I am not among them. I favor efforts to make sense of our lives from the standpoints of many different fields–including philosophy, literature, mathematics, theology, languages, statistics, physics, psychology, and more.  Each field contributes in some way to our understanding.

But how does one work with so much uncertainty? First of all, enjoy it now and then. It would be a dreary world if we could figure everything out. One doesn’t have to be perpetually cheery about it, but one can take courage from it. Second, find ways of working with degrees of uncertainty–not treating all uncertainty as alike, but determining which are greater than others. Models in the social sciences can bring much insight; one must just take care to observe their divergence from actual phenomena.

It’s important, when doing this, to avoid the null hypothesis fallacy. Some might say, “Well, I can’t prove a theory true, but I can prove its negation false, and that’s essentially evidence for the theory.” No, it isn’t. The two are not the same. When proving the negation false, one does not win evidence for the theory; one is still firmly fixed in the wobbles of doubt.  One must figure out how to view the doubt clearly.

In any case, the answer to the initial question is double. If you are working within a closed system, you can prove a theory (or theorem) true; within an open system, you cannot. However, in the latter case there is still much you can learn.

A mini-glossary:

Hypothesis: A proposed explanation for the way something works. (It is more than a “guess”; it must have a basis in evidence and reasoning, and it must be testable.)

Theory: A hypothesis that has been tested, substantiated, and extended, and that applies broadly to a natural phenomenon or class of phenomena.

Law: Like a theory, but unified into a general principle that unerringly explains a phenomenon (to the best of our knowledge and understanding).

Model: A representation of a real-world phenomenon, designed to assist with observation, testing, and explanation.

Note: I revised this piece substantially after posting it. In particular, I clarified the terms, changed the examples, and added some links. I cut the part about literature, since it needs a post of its own.

Interesting Studies with Hasty Conclusions

I recognize that I was a bit harsh on the calendar synaesthesia study–that is, dismissive in tone. What bothered me was the claim (right there in the paper itself) that this constituted the first “clear unambiguous proof for the veracity and true perceptual nature” of calendar synaesthesia. I sincerely thought, for a little while, that this might be a hoax.

The experiments of the study do not prove anything, nor do they have to. It would be far more interesting (to me) if the authors explored the uncertainties a bit more.

For instance, when you have a mitigating conditioned response, how do you tell what is synaesthesia and what isn’t? Say, for instance, that you have learned to read sheet music from a young age. Suppose, now, that when you hear music, you see musical notes before you. Is this synaesthesia, or is this a learned association between musical notation and sounds?

The calendar situation seems similar. We have all seen calendars (many times). They take different shapes but always arrange the dates in some pattern. In addition, we have seen clocks, season wheels, and other representations of time. If I can picture the months in an atypical (fixed) shape, is this synaesthesia, or is it a modification of learned associations between time and images?

I do not doubt the existence of synaesthesia (of certain kinds). I just see reason to try to delineate what it is and isn’t–and to refine the surrounding questions, including questions of methodology.

The other day, Andrew Gelman posted an (exploratory) manifesto calling for more emphasis  on–and better guidelines for–exploratory studies. The piece begins with a quote from Ed Hagen:

Exploratory studies need to become a “thing.” Right now, they play almost no formal role in social science, yet they are essential to good social science. That means we need to put as much effort in developing standards, procedures, and techniques for exploratory studies as we have for confirmatory studies. And we need academic norms that reward good exploratory studies so there is less incentive to disguise them as confirmatory.

The problem is twofold: (1) Exploratory studies don’t get enough respect or attention, so people disguise them (intentionally or not) as confirmatory studies; (2) Exploratory stories can be good, bad, and anything in between, so there should be clearer standards, procedures, and techniques for them. “Exploratory” does not (and should not) mean “anything goes.”

But then comes the question: What constitutes a good exploratory study? Gelman offered a few criteria, to which others added in the prolific comment section.

It would be encouraging if the social sciences (and other fields, including literature) worked carefully with uncertainties (and got published because they did so). Instead of iffy studies, we’d have studies that wielded the “if” with skill and care.

P.S. Speaking of Andrew Gelman’s blog, I had some fun responding to Rolf Zwaan’s “How to Cook Up Your Own Social Priming Article.”

 

Calendar Synaesthesia Hoax (I Wish)

hulaI wish it were a hoax, because then I could cachinnate without guilt. As it is, I still laugh, but with trouble in the belly. I am sorry about the gullibility in the world.

I learned about it from Cari Romm’s piece in New York Magazine. The title grabbed me: “There’s a Form of Synesthesia Where People Literally See Time in Front of Them.” I thought: That’s quite something, seeing time! I imagined some kind of visual perception of a non-spatial continuum of events. Some sort of visible yet invisible flow.

Instead, the “calendar synaesthete”–one subject in a study with eight controls–could picture the months of the year in geometrical arrangement. For this subject, they took a V shape;  for a subject of a previous experiment, the shape of a hula-hoop.

The authors call their paper (published in Neurocase) the first “clear unambiguous proof for the veracity and true perceptual nature” of calendar synaesthesia. Really? This got published in Neurocase and reported in New Scientist and New York Magazine?

Synaesthesia (also spelled “synesthesia”) is the name for what happens when an event that stimulates an experience in one sensory or cognitive pathway also stimulates it in a second (and unexpected) one. For instance, some synaesthetes see sounds, associate letters of the alphabet with specific colors, or smell numbers. The phenomenon exists. But does this particular study tell us anything about it?

This is one of several experiments that led to their “clear unambiguous proof” (the quote below is from the New Scientist article):

Next they asked ML and eight non-synaesthetes to name the months of the year backwards, skipping one or two months each time – a task most people find challenging. They figured that ML should be able to complete the task quicker than the others as she could read it from her calendar. Indeed, ML was much quicker at the task: when reciting every three months backwards, she took 1.88 seconds per month compared with 4.48 seconds in non-synaesthetes.

First of all, what does any of this have to do with visualizing time? From what I can tell, it’s about recalling and manipulating the sequence of months. There may or may not be a visual component in such calculation; either way, this experiment shows no synaesthesia per se. Second, who takes 4.48 seconds to recite every third month backwards? I can do it in under 2 seconds per month, without seeing any V shape, donut, hula-hoop, or Moebius strip.

Here’s what the paper says:

In control subjects, the average RT for reciting all of the months backward (n = 8) was 1.46 s/month. For skipping 1 or 2 months – the average was 2.54 and 4.48 s/month respectively. For ML, the average RT for the same 3 tasks were (A) 0.58 s/month, (B) 1.63 s/month, and (C) 1.88 s/month (see legends in Figure 2).

There were eight controls and one subject. Yes, just one. (Nor does the study explain how the subject and controls were selected.) Their study of  a second subject, HP, was incomplete: “We then studied the second subject – HP – but for practical reasons – were only able to conduct a subset of the experiments that we had performed on ML.” (She was able to recite the months as quickly as ML, though.)

To supplement the findings, perhaps, they mention EA, a subject from a previous study:

Indeed, on a previous occasion, we had informally tested a synesthete EA, who might have qualified as a higher calendar synesthete. Her calendar form was shaped like a hula-hoop (the most common manifestation of calendar forms) in the transverse plane in front of her chest. Unlike ML, though, when EA turned her head rightward or leftward, the calendar remained stuck to the body, suggesting that it was being computed in body-centered, rather than head (and eye) centered coordinates. The variation across calendar synesthetes, in this regard, reminds us that even in neurotypical brains there are probably multiple parallel representations of body in space that can be independently accessed depending on immediate task demands.

How did they get from the hula-hoop to “multiple parallel representations of body in space”–and from any of this to “clear unambiguous proof” of the existence of calendar synaesthesia?

I do not doubt that people can picture calendars; people can picture all sorts of things, and calendars are already visual representations of a model of time. I see no synaesthesia in the ability to picture something that is already a picture.

I recognize that this is the authors’ very point: that for this subject, the calendar  is something more than a strong mental picture. Yet the experiments do not prove this.

Note: I made some revisions and additions to this piece after posting it–and deleted one sentence that in retrospect seemed excessively sarcastic. Also see Shravan Vasishth’s comment and my response. I may have been too caustic overall–but I hold to my view that the researchers went too far in declaring “proof.” See my followup post.

Happiness Surveys Actually Increase Happiness

Happiness surveys are all the rage these days–but did you know that they can make you happier? Such is the finding of a research group at the Wisconsin Institute for Scientific and Demographic Organizational Measurement. The study, currently under peer review, stands out as the most robust and extensive investigation of the question to date.

Felix Laimingas, the lead researcher and a professor of brain eudaimonia, explained the methodology over sea salt toffee bars and tea. “We gave happiness surveys to a random sampling of 500 pedestrians in Milwaukee, Madison, and Green Bay,” he said. “For the control group, we approached random pedestrians and asked them, ‘How are you doing?'”

Those who completed the one-question survey (145 out of the original 500) gave their happiness level a mean rating of 7.2 out of 10 (95% CI: 6.8–7.6), whereas the control group reported a mean happiness level of just slightly over “okay,” which translates to 5.2 (95% CI: 4.9–5.5). The difference between the happiness levels of the survey takers and the control group is statistically significant at 20%.

What are the implications? “Well, we’ve got two things to think about,” said Laimingas. “First, since happiness surveys are actually making people happier, they might be affecting some of the research out there on happiness. That’s not a big concern, though, because really all they’re doing is lifting the  level across the board, leaving comparisons intact.” He paused as we each took another sea salt toffee bar, leaving none behind. “The other implication is even more uplifting. If we do lots more of this happiness survey type thing, we’ll make the world a better place. This is TED talk material.”

I expressed some doubt, since my own action research has found that healthy skepticism increases my sense of agency, which (up to a point) makes me happier. Laimingas smiled, being a happy person himself (with a nice personality type to boot). “I’m glad you questioned our findings,” he said. “That gives me a chance to prove them to you.” He offered me a survey and a pencil.

The survey consisted of two questions:

  1. How happy are you now, on a scale of 0 to 10?
  2. How happy are you now, on a scale of 0 to 10?

The first time, I answered “6”; I was fairly content with my life but didn’t want to be naive about the matter. The second time, I gave the same answer.

“Your happiness actually went up,” said Laimingas. “You see, we have to consider the law of repetitive decay. Research has shown that when survey respondents answer the same question twice in a row, they’re likely to be disillusioned or bored the second time. You stayed right at the same level, which means you counteracted the tendency to go down. This means that you were happier as a result of answering these questions.”

I was happy to hear this, since I initially worried (productively) that my duplicate numbers might come across as rude or flippant. “So you are happy with my answers?” I asked.

“Oh, very happy. Happier than ever. And validated.”

If there is bias in happiness, I thought, it’s good bias. Maybe we should have more happiness  studies. Our takeaway: The next time you verge on asking someone “How are you?” consider handing out a quick survey instead. It’s good for science and the world.

“The peacock spreads his fan”

I learned about Leonard Cohen’s death from Virgil Shaw, who mentioned it in between songs last night, during a superb show. I didn’t check my phone (and the news) until later, but there it was. Leonard Cohen is gone. Is that true? Is he gone? His music is playing in my mind, so he isn’t gone; the songs carry on in his place. What’s hitting me, though, is the knowledge  that his work is now sealed, that there will be no more new songs. Even more than that, it’s the knowledge that the person who wrote “Suzanne,” “Story of Isaac,” “Avalanche,” “The Stranger Song,” “Dance Me to the End of Love,” and “Hallelujah” is no longer here. Even there, it’s hard to pinpoint the sadness. He could have died earlier or later; maybe he could have lived until a hundred. At some point he would have had to go. Nor would I ever have met him, as far as I know, nor does that have anything to do with the tightness in my throat right now. What hurts is the loss of a fighter for language and song, who I trusted was somewhere breathing.

Note: I made minor revisions to this piece after posting it. It was hard to get the words right. I commented on the New York Times obituary as well; see the many beautiful comments  there.

Update: See Leon Wieseltier’s moving eulogy.

Can Happiness Be Rated?

pandaFirst, I’ll upend a possible misunderstanding: My point here is not that “so many things in life cannot be measured.” I agree with that statement but not with the abdication surrounding it. It is exquisitely difficult to measure certain things, such as happiness, but I see reason to peer into the difficulty. Through trying and failing to measure happiness, we can learn more about what it is.

Lately I have seen quite a few studies that include a happiness rating: the study I discussed here, a study that Drake Baer discussed just the other day, and a study that Andrew Gelman mentioned briefly. In all three, the respondents were asked to rate their happiness; in none of them was happiness defined.

Some people may equate happiness with pleasure, others with contentment, others with meaning. Some, when asked about their happiness level, will think of the moment; others, of the week; still others, of the longer term. The complexities continue; most of us are happier in some ways than in others, so how do we weigh the different parts? The weights could change even over the course of a day, depending on what comes into focus. Happiness changes in retrospect, too.

In addition, two people with similar “happiness levels” (that is, who would describe their pleasure, contentment, and meaningful pursuits similarly) might choose different happiness ratings. A person with an exuberant personality might choose a higher rating than someone more subdued, or vice versa.

Given the extraordinary complexity of measuring happiness, I distrust any study that measures it crudely and does not try to define it. I doubt that it can be defined or measured exactly; but a little more precision would be both helpful and interesting.

Incidentally, the search for precision can bridge the humanities and the sciences; while they will always have different methodologies (and even different questions), they have a common quest for the right words.

One Nation Not on Speaking Terms with Itself

As I woke up intermittently throughout the night and disbelieved myself into the news,  I realized that I knew of only one person (among friends, acquaintances, family, relatives, and colleagues) who may have voted for Trump. There may be more; I’m just not aware of them.

How much do we talk to each other in this country? I don’t just refer here to people from Vermont talking with people from Kansas. I refer to the people in our  neighborhoods, families, professions, and so on. I am not that talkative overall, so there are many with whom I never talk daily. Why would those happen to include just about everyone who voted for Trump?

The “sorting” phenomenon in the U.S. is well known, documented, and discussed; people tend to associate with those who generally agree with them. Sure, some disagreements come up, but often the big ones stay out of the picture, to the point where we simply don’t know anyone who views things in a markedly different way.

I have generally thought of myself as a semi-outsider, as someone who stays out of gossip and chatter and holds somewhat unconventional views. But when I look from a distance, I see that I “fit in” with my environment more than not. I do not have daily encounters with people who believe that I am going to hell because I read things other than the Bible (or because I read the “wrong” parts of the Bible). I have never (to my knowledge) hobnobbed with a creationist, at least not about creationism. Only a few times have I discussed immigration with people who feel that the U.S. should close its gates. Except for a few artists, writers, and entrepreneurs, I know no one who sees college as a waste of time. In addition, I know few U.S.-born poor and working-class Caucasians. Nearly all my friends, family, colleagues, and acquaintances are middle-class, immigrant, non-Caucasian, or some combination of the three.*

But the insulation goes a bit deeper. I suspect that we know as little about ourselves and those close to us as we know about those from the “other side” of the political divide. If it’s true that we know ourselves at least partly through dialogue, then our self-knowledge depends on the quality and extent of such dialogue. If we are limited to certain topics, certain people, certain moods, and certain assumptions, then a lot will remain unexamined.

My response to the election is to look at my conversations. With whom am I talking, and with whom not? Which ideas and topics come up, and which do not? It is worthwhile to live in the uncertainty of this: to listen beyond the familiar sounds.

Note: I made a few edits and additions  to this piece after posting it. Also, see today’s excellent pieces by Jesse Singal and Drake Baer.

*Added after the initial additions and revisions.

“Took the Northern Northern”

northern-northern

It has been an eventful past 24 hours. After finishing the take-home essay for my cantillation midterm, I went last night to hear the Dessoff Choirs perform a fantastic concert, at Alice Tully Hall, of Steven Stucky’s Take Him, Earth and Whispers, David Hurd’s In Honor of Martin (the world premiere of the orchestrated version), and Mozart’s Requiem.

In the morning I reviewed once more for the midterm, went and took it, walked over to Columbia Secondary School, talked for a while with the principal and a few others, came back home, and went to vote. Voted.

Later this week I will get to hear 20 Minute Loop play the record release show for their new album, Songs Praising the Mutant Race. The blog post title and photo are in honor of the phrase “took the Northern Northern” from their song “Hell in a Handbasket.”

I think about how a blog seems to give a sense of what a person is up to, but actually does not, or might not, more than fractionally. That got me to thinking about how my favorite literature tends to reveal how little we know about ourselves and others. There’s an illumination and humility in it. Such revelation can’t follow a formula; it takes you by surprise again and again, because of the way it stands out against the world, bursts from its frame, and lifts you up in its arms.

Beyond the Introvert-Extravert Divide

Over at New York Magazine, Drake Baer has been challenging the introvert-extravert dichotomy with vigor. “‘Introvert or Extrovert’ Is the Wrong Way to Define Your Identity,” declares one October article; an article from July has a similarly bold title (“Why Declaring ‘I’m an Introvert!’ Limits Your Life“). In both articles, and in some earlier pieces, Baer emphasizes the complexity of personality and the influence of occupation and context. I would go even farther than he does—for instance, I am skeptical of the Big Five theory of personality—but I applaud his combination of boldness and subtlety.

The introvert issue has been so overhyped that it swept other discussions into its hot air. It created a “groupthink” of its own. In 2012, a few months after Republic of Noise came out, I was interviewed for an Education Week article on introverts in the classroom (as was Susan Cain). When speaking with Sarah Sparks, I emphasized the distinction between solitude and introversion. Solitude is essential to education (in some way and in some form) no matter what your personality type. Instead of trying to make the classroom amenable to introverts (who are a highly diverse bunch, with a wide range of preferences and needs), pay attention to the subject matter. It just isn’t true that “introverts” prefer online discussion to class discussion. If you are approaching the subject keenly, your class discussion will not be dominated by table-thumping loudmouths anyway. People will have to think, because there will be something to think about. Of course you should pay attention to the students—but for their ideas and unique qualities, not their type.

But these points were left out of the article;  Sparks and other reporters continued to present issues in terms of introverts and extraverts. I have wondered why. It seems part of our country’s tendency toward polarization. It isn’t so far removed, in other words, from the climate of the election. It is all too easy to identify yourself with an oppressed group (in this case the introverts) and let someone else tell you who  you are and what you need. Someone shows up who seems to tell your story, explains how you and your kind have been mistreated, and promises a revolution.

But maybe this isn’t quite your story; maybe your personal oppression (to the extent that it exists) comes from many places, including the self; maybe liberation lies not in an uprising of your personality type but in good independent thought. I don’t mean that one should reject all alliances, but no alliance should demand a reduction of the mind or soul. There should be room to challenge not only the dominant train of thought but its underlying suppositions. There should be room to say, “this isn’t quite right.”

I see Baer’s articles as a promising step in that direction. A shout-out to Melissa Dahl too.

Note: I originally mistitled the first Baer article; the error is now fixed.