I recognize that I was a bit harsh on the calendar synaesthesia study–that is, dismissive in tone. What bothered me was the claim (right there in the paper itself) that this constituted the first “clear unambiguous proof for the veracity and true perceptual nature” of calendar synaesthesia. I sincerely thought, for a little while, that this might be a hoax.
The experiments of the study do not prove anything, nor do they have to. It would be far more interesting (to me) if the authors explored the uncertainties a bit more.
For instance, when you have a mitigating conditioned response, how do you tell what is synaesthesia and what isn’t? Say, for instance, that you have learned to read sheet music from a young age. Suppose, now, that when you hear music, you see musical notes before you. Is this synaesthesia, or is this a learned association between musical notation and sounds?
The calendar situation seems similar. We have all seen calendars (many times). They take different shapes but always arrange the dates in some pattern. In addition, we have seen clocks, season wheels, and other representations of time. If I can picture the months in an atypical (fixed) shape, is this synaesthesia, or is it a modification of learned associations between time and images?
I do not doubt the existence of synaesthesia (of certain kinds). I just see reason to try to delineate what it is and isn’t–and to refine the surrounding questions, including questions of methodology.
Exploratory studies need to become a “thing.” Right now, they play almost no formal role in social science, yet they are essential to good social science. That means we need to put as much effort in developing standards, procedures, and techniques for exploratory studies as we have for confirmatory studies. And we need academic norms that reward good exploratory studies so there is less incentive to disguise them as confirmatory.
The problem is twofold: (1) Exploratory studies don’t get enough respect or attention, so people disguise them (intentionally or not) as confirmatory studies; (2) Exploratory stories can be good, bad, and anything in between, so there should be clearer standards, procedures, and techniques for them. “Exploratory” does not (and should not) mean “anything goes.”
But then comes the question: What constitutes a good exploratory study? Gelman offered a few criteria, to which others added in the prolific comment section.
It would be encouraging if the social sciences (and other fields, including literature) worked carefully with uncertainties (and got published because they did so). Instead of iffy studies, we’d have studies that wielded the “if” with skill and care.