A Statistical Study of Literature

e-h-_shepard_illustration_of_mr_toadRecently Drake Baer reported on a study of the emotional arcs of stories. Working from ideas that Kurt Vonnegut puts forth in his master’s thesis and 1985 lecture, a research group, led by University of Vermont Ph.D. candidate Andrew J. Reagan, analyzed the progression of happiness levels in a selection of 1,327 stories. They did this by measuring the “happiness levels” of the words themselves and mapping the movement in happiness over the course of the story.

They found six basic shapes. In Baer’s words, “A full 85 percent of the books analyzed fell under one of six shapes. They are: ‘rags to riches,’ in which sentiment goes up; ‘riches to rags,’ where it goes down; ‘man in a hole,’ in which there’s a fall, then a rise; ‘Icarus’, where it’s a rise then a fall; ‘Cinderella,’ or rise-fall-rise; and ‘Oedipus,’ or fall-rise-fall.” It isn’t the plot that matters here, Baer explains, but “the way sentiment [is] conveyed by the words themselves, and whether they rise or fall, signaling happy or sad endings to chapters and sections and books as a whole.”

The initial problem is that the arcs don’t really match the stories themselves. Here’s the arc that supposedly matches the emotional arc of The Wind in the Willows by Kenneth Grahame (one of my favorite books from childhood):

13688_2016_93_fig4_htmlThis looks like what the authors call an “Oedipus” pattern, or a “fall-rise-fall.” But the ending of The Wind in the Willows doesn’t come across as a “fall” at all. There’s so much irony and humor in it that you can’t call it one thing or another–but Toad does come out victorious, in a subtle way. He has escaped from prison, has made a triumphant return to Toad Hall, and is about to have a banquet in his name. Glory awaits–but the Badger and the Rat thwart his intention to deliver a speech, an address, a song, and other compositions (all composed by Toad himself). Instead, they impress upon him that it’s time for him to turn a new leaf.

So what does he do? Before the banquet, he retires to his bedroom, takes to “giggling in a shy, self-conscious manner,” arranges all the chairs in a semicircle, and “[takes] up his position in front of them, swelling visibly.” For this invisible audience he performs his Final Song, “with great unction and expression”; once finished, he does it over again.

Then he enters the banquet–and when the animals cheer and congratulate him, he murmurs, “Not at all!” and “On the contrary!” They all marvel at his change of attitude, not realizing that he is giving his greatest performance yet. He is fully aware of the effect: “At intervals he stole a glance at the Badger and the Rat, and always when he looked they were staring at each other with their mouths open; and this gave him the greatest satisfaction.”

Then there’s a delightful epilogue in which they live “in great joy and contentment, undisturbed by future risings or invasions.” Some little twists in the last two paragraphs keep the story ticklish; it isn’t just a tale of a reformed and sobered Toad. In some ways he will always be Toad, ludicrous and lovable.

So, what went wrong with the hedonometer? Here are some possible pitfalls of the researchers’ approach.

  1. The arc of a story is not only emotional; it also involves action, understanding, form, and language. By isolating the emotion, you can distort the shape of the story.
  2. If you do choose to consider the emotional arc on its own, you must look beyond “happiness” levels. There’s much more to a story’s emotion than its apparent happiness.
  3. If you do choose to look at happiness levels, you must look beyond the “happiness counts” of the words themselves. Part of the point of literature is to convey things that cannot easily be conveyed through  other modes. To convey happiness, literature often uses images, rhythms, and other indirect representations.
  4. Mixtures of emotion in literature have meanings of their own. They do not average out to in-between emotions.
  5. When conducting a study of literature, you have to know the literature in question. You must be in a position to determine whether your findings actually make sense.

I do not fault the researchers for conducting this experiment. It’s interesting on the face. It’s especially interesting to see where and why it can go wrong. As happens so often, the researchers do not subject their findings to sufficient doubt. Their analysis gets especially shaky when they discuss “the success of stories”:

To examine how the emotional trajectory impacts success, in Figure 7 we examine the downloads for all of the books that are most similar to each SVD mode (for additional modes, see Figure S3 in Appendix B in Additional file 1). We find that the first four modes, which contain the greatest total number of books, are not the most popular. … We find ‘Icarus’ (-SV 2), ‘Oedipus’ (-SV 3), and two sequential ‘Man in a hole’ arcs (SV 4), are the three most successful emotional arcs. These results are influenced by individual books within each mode which have high numbers of downloads, and we refer the reader to the download-sorted tables for each mode in Appendix E in Additional file 1.

If the arcs correspond only weakly to what is going on in the story, the number of downloads corresponds only weakly to a story’s “success.” Moreover, as the authors themselves acknowledge, the numbers are influenced by particular books.

This all points to the importance of subject-matter knowledge in research. It’s fine to subject literature to statistical experiments–but then test your findings against the works themselves. Also, question, question, question! With more questioning, this study could lead to insights into the possibilities and pitfalls of statistical analyses of literature.

(Illustration by E. H. Shepard.)

Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s