Against Science by Pun
I’ve been preparing for a first-year screener exam as part of my PhD program. This involves reading some 110 papers published in the Journal of Consumer Psychology between 2019 and 2020. At this point, I have a good sense of what work is being done in the field — and what practices should end. Science by Pun is one of them.
If I were braver, I would include a list of papers published in the Journal of Consumer Research that exemplify what I consider a very bad practice. However, I would like to get a job when I graduate. I will therefore limit myself to criticizing a finding that has already been picked apart by others, in a field one step removed from my own.
Consider the following statement:
“Holding warm coffee makes people emotionally warmer”
The statement is a reference to the work of John Bargh, a Yale psychologist, who is well known for his work on priming. He has also written a book on the subject, titled “Before You Know It: The Unconscious Reasons We Do What We Do”. This book is filled with all the apparently unconcious reasons humans have for their behaviors. Another quirky example is Bargh’s assertion that drinking warm soup can treat depression, stating that “the warmth of the soup helps replace the social warmth that may be missing from the person’s life”.
The warm coffee study was published in Science, one of the most prestigious journals in the world. For many, a publication in Science is the peak of their career. So what does it take to get such a claim published? Not a lot, apparently. The main study in that paper had 41 participants split between 2 conditions (warm coffee, iced coffee). The authors found that people holding warm coffee rated a stranger as 11% warmer on a 7-point scale ranging from Cold to Warm. There are many things I could say about this, but suffice to say that perhaps 41 Yale undergraduates do not exactly represent all of humanity, and we should have a little humility when drawing large implications from any study limited to such a population.
These sorts of claims fail to replicate, and in my opinion are symptoms of a larger problem in social science research. Social scientists want flashy findings that get them book deals, ted talks, and glowing NYT reviews, and this pushes them to sensationalize their own research, to the detriment of the rest of society.
“Drinking warm coffee makes people emotionally warmer” is a clear example of pun-based research. Authors who make such claims are taking advantage of the ambiguous definition of the word “warm”, which can apply to temperature OR personality, and using that to tie together seemingly disparate lines of research. And can you even blame the researcher? Finding a snazzy title for a research project has many benefits! It can 1) make working on the project more fun, and 2) help market your findings to peer reviewers and the broader public.
The increase in marketability may serve to offset a lack of methodological rigour, or lend itself to the generation of public hype disproportionate to the actual findings of the research. I believe this to be the case with the Warm Coffee research (and likely other findings like “drinking soup treats depression”).
This is particularly bad because methodological standards in psychology are so lax that virtually any finding can be demonstrated with sufficient motivation on the part of the researcher. Questionable Research Practices abound, statistical power is depressingly low, and authors routinely exaggerate their otherwise rigorous (or at least, not horribly bad) work when given a public platform.
Given how easy it can be to get the finding you want, I would bet a lot of money that some researchers are making puns first, designing studies second.
I worry that many papers in my own field follow a similar pattern. Some compelling string of words is generated that, if true, may represent a novel contribution to the field. However, such a research project would never be pursued but for the semantic relationship between two seemingly disparate activities.
For fun, I’ve thought of some examples:
- People with empty social lives prefer fuller drinks
- People are more dominant over customer service workers when holding their phone in their dominant hand
- Drinking beer with lots of “hops” improves physical prowess (specifically, running long-jump)
- People in warm environments perceive customer service representatives as warmer
If anyone’s interested in researching these very important phenomena, get in touch.