I have been thinking a lot about formal and informal research: how both have a place, but how they shouldn’t be confused with each other. One of my longstanding objections to “action research” is that it confuses the informal and formal.
Andrew Gelman discusses this problem (from a statistician’s perspective) in an illuminating interview with Maryna Raskin on her blog Life After Baby. It’s well worth reading; Gelman explains, among other things, the concept of “forking paths,” and acknowledges the place of informal experimentation in daily life (for instance, when trying to help one’s children get to sleep). Here’s what I commented:
[Beginning of comment]
Yes, good interview. This part is important too [regarding formal and informal experimentation]:
So, sure, if the two alternatives are: (a) Try nothing until you have definitive proof, or (b) Try lots of things and see what works for you, then I’d go with option b. But, again, be open about your evidence, or lack thereof. If power pose is worth a shot, then I think people might just as well try contractive anti-power-poses as well. And then if the recommendation is to just try different things and see what works for you, that’s fine but then don’t claim you have scientific evidence one particular intervention when you don’t.
One of the biggest problems is that people take intuitive/experiential findings and then try to present them as “science.” This is especially prevalent in “action research” (in education, for instance), where, with the sanction of education departments, school districts, etc., teachers try new things in the classroom and then write up the results as “research” (which often gets published.
It’s great to try new things in the classroom. It’s often good (and possibly great) to write up your findings for the benefit of others. But there’s no need to call it science or “action research” (or the preferred phrase in education, “data-driven inquiry,” which really just means that you’re looking into what you see before you, but which sounds official and definitive). Good education research exists, but it’s rather rare; in the meantime, there’s plenty of room for informal investigation, as long as it’s presented as such.
[End of comment]
Not everything has to be research. There’s plenty of wisdom derived from experience, insight, and good thinking. But because research is glamorized and deputized in the press and numerous professions, because the phrase “research has shown” can put an end to conversation, it’s important to distinguish clearly between formal and informal (and good and bad). There are also different kinds of research for different fields; each one has its rigors and rules. Granted, research norms can also change; but overall, good research delineates clearly between the known and unknown and articulates appropriate uncertainty.
Update: See Dan Kahan’s paper on a related topic. I will write about this paper in a future post. Thanks to Andrew Gelman for bringing it up on his blog.