Interesting Studies with Hasty Conclusions

I recognize that I was a bit harsh on the calendar synaesthesia study–that is, dismissive in tone. What bothered me was the claim (right there in the paper itself) that this constituted the first “clear unambiguous proof for the veracity and true perceptual nature” of calendar synaesthesia. I sincerely thought, for a little while, that this might be a hoax.

The experiments of the study do not prove anything, nor do they have to. It would be far more interesting (to me) if the authors explored the uncertainties a bit more.

For instance, when you have a mitigating conditioned response, how do you tell what is synaesthesia and what isn’t? Say, for instance, that you have learned to read sheet music from a young age. Suppose, now, that when you hear music, you see musical notes before you. Is this synaesthesia, or is this a learned association between musical notation and sounds?

The calendar situation seems similar. We have all seen calendars (many times). They take different shapes but always arrange the dates in some pattern. In addition, we have seen clocks, season wheels, and other representations of time. If I can picture the months in an atypical (fixed) shape, is this synaesthesia, or is it a modification of learned associations between time and images?

I do not doubt the existence of synaesthesia (of certain kinds). I just see reason to try to delineate what it is and isn’t–and to refine the surrounding questions, including questions of methodology.

The other day, Andrew Gelman posted an (exploratory) manifesto calling for more emphasis  on–and better guidelines for–exploratory studies. The piece begins with a quote from Ed Hagen:

Exploratory studies need to become a “thing.” Right now, they play almost no formal role in social science, yet they are essential to good social science. That means we need to put as much effort in developing standards, procedures, and techniques for exploratory studies as we have for confirmatory studies. And we need academic norms that reward good exploratory studies so there is less incentive to disguise them as confirmatory.

The problem is twofold: (1) Exploratory studies don’t get enough respect or attention, so people disguise them (intentionally or not) as confirmatory studies; (2) Exploratory stories can be good, bad, and anything in between, so there should be clearer standards, procedures, and techniques for them. “Exploratory” does not (and should not) mean “anything goes.”

But then comes the question: What constitutes a good exploratory study? Gelman offered a few criteria, to which others added in the prolific comment section.

It would be encouraging if the social sciences (and other fields, including literature) worked carefully with uncertainties (and got published because they did so). Instead of iffy studies, we’d have studies that wielded the “if” with skill and care.

P.S. Speaking of Andrew Gelman’s blog, I had some fun responding to Rolf Zwaan’s “How to Cook Up Your Own Social Priming Article.”

 

Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: