The Future of Academic Psychology

Or: The birth of a second “dismal science”

In my last post on Tal Yarkoni’s generalizability crisis, I outlined a radical proposition: that the imprecision and statistical incoherence of psychologists’ verbal claims makes our research more akin to performance art than other sciences. Given that our current theories are heavily underdetermined by their supporting evidence, it may be more accurate that psychological research — at least that which goes on in the lab — is more about finding contexts in which a theory is correct than investigating the validity of theories themselves. We choose a theory we like, and put on a show to prove it — going through the ancient and solemn tradition of establishing that p < 0.05 in the process.

I can’t get this framing out of my head, and want to reconcile it with the idea that I’m starting a PhD in psychology (consumer behaviour) — in fact, I’m writing this the day of my orientation.

Yarkoni gives us two options: do science better, or engage in performance art. I want to look at what doing science better might look like, and frame it in a way I can accept on an emotional level.

Two paths

I’ve already explored what psychology as performance might look like, and have, to date, spent most of my time on that idea. I’ll now try to go deeper into the second option: actually doing science better.

From my last essay:

• Draw more conservative inferences. Don’t overextrapolate. Define the “boundaries of the universe of observations the authors believe their findings apply to”
• Take descriptive research more seriously. Why do we look down on descriptive research? It’s useful, it’s interesting, and it’s way more solid than intervention-type research.
• Fit more expansive statistical models. Begin accounting for all the sources of variance that psychology has, to date, ignored. Adding random effects is just the beginning.
• Design with variation in mind. Controlling things as tightly as possible makes it incredibly hard to generalize. Design experiments that enhance variance and account for it. This will take more resources but could be worth it.
• Emphasize variance estimates. Rely less on point estimates and instead put more focus on the range of possible effects. Start looking at within and between-study variance of effect instead of just point estimates. This is not to say people don’t already do this, but they should do it more.
• Make riskier predictions. Too often our studies are on tiny effects that would build on a greater literature and in aggregate may support a theory. We should instead subject our theories to “severe tests”, or tests that, depending on the outcome, could truly give strong evidence in support or against our pet theories.
• Focus on practical predictive utility. A favourite phrase of mine is “make your beliefs pay rent in anticipated experiences”. Instead of asking “does verbal overshadowing exist”, instead ask: can we train a statistical model that allows us to meaningfully predict people’s behaviors in a set of situations that superficially seem to involve verbal overshadowing?

To summarize an already oversimplified summary: be more conservative and don’t make life easy on yourself; go for big questions that would constitute significant challenges to established theories, and have a better sense of your own limitations as a researcher in controlling for sources of variance.

While psychologists may claim they “follow the data” already — there’s a lot more to be done in order to achieve this ideal. If psychology is currently performance art in how heavily “scripted” it seems, what would it look like as a less scripted enterprise? What if it looked less like theatre… and more like improv?

Improv Psychology

Keith Johnstone is a former university professor and professional theorist of improv. He wrote about his particular philosophy of improv in a book title Impro, and one line has stuck with me:

“The improviser has to be like a man walking backwards. He sees where has has been, but pays no attention to the future. His story can take him anywhere, but he must still ‘balance’ it, and give it shape, by remembering [shelved incidents].”

This is not dissimilar from how Yarkoni would have us do science better. Why should science work like a plot with a set story? When researchers “know” the end result of their studies and contort the research (read: plot) to arrive at that result, we call that p-hacking, fraud, bias, and a number of other invectives according to their particular method of satisfying their desired narrative arc.

Science as improv matches far better with Yarkoni’s suggestions. Like an improv actor, we should follow the data where it leads — incorporating past results, theories, and analytical tools when appropriate. I’ve marshalled the best quality data and assorted anecdotal accounts to construct the following peer-reviewed representation of psychology’s choice:

An aesthetic representation of the two choices psychology researchers face.

In Yarkoni’s estimation — and my own — research psychologists are in a weird state where they’ve fooled themselves into thinking they’re “following the data” more than they actually are. Like my beautiful imagery above, there’s a disconnect from their imagined position on the unscripted-scripted axis and what they do in their day to day. The two options to follow are at either end of that axis — either Yarkoni’s hyper-conservative, boring science, or the exciting performance art a half-step from our current state.

This is obviously contrived, but so was the original framing of psychology as performance art. The point here is not to formulate a direct equivalence between academic psychology and improv, but to leverage existing concepts of improv and explore how they might apply to our research practices. In this, I think the concept succeeds. Thinking of psychology’s future as a discipline more similar to improv than scripted theatre is something I expect to carry with me for a long time.

The second dismal science

Or: why this new psychology isn’t quite like improv.

This new psychology would differ from its current form in that it would get a lot more boring. Everyone wants to read the latest “standing near garbage increases homophobia by 50% paper”, but this new psychology would lose the pizzazz of the old priming studies.

They may be right — but are their claims justified by the data alone? Source

When asked for their opinions on political issues of the day, we may see less psychologists overextending their research to favour their preferred party, and more answers amounting to “it depends” or “I don’t know”. While modern day psychologists are loved by journalists for their fantastical (and convenient) claims, this new psychology would lose much that made it a media darling. This reminds me a lot of how economics has fared in the public sphere.

Economics is famously known as the “dismal science”. Most people assume it’s because economics is boring, but there’s a more interesting history to that term. From David Levy and Sandra Peart:

“Everyone knows that economics is the dismal science. And almost everyone knows that it was given this description by Thomas Carlyle, who was inspired to coin the phrase by T. R. Malthus’s gloomy prediction that population would always grow faster than food, dooming mankind to unending poverty and hardship.

While this story is well-known, it is also wrong, so wrong that it is hard to imagine a story that is farther from the truth. At the most trivial level, Carlyle’s target was not Malthus, but economists such as John Stuart Mill, who argued that it was institutions, not race, that explained why some nations were rich and others poor. Carlyle attacked Mill, not for supporting Malthus’s predictions about the dire consequences of population growth, but for supporting the emancipation of slaves. It was this fact–that economics assumed that people were basically all the same, and thus all entitled to liberty–that led Carlyle to label economics “the dismal science.”

Carlyle disagreed with the conclusion that slavery was wrong because he disagreed with the assumption that under the skin, people are all the same. He argued that blacks were subhumans (“two-legged cattle”), who needed the tutelage of whites wielding the “beneficent whip” if they were to contribute to the good of society.” — David Levy and Sandra Peart (Source)

Economics became the dismal science, not for its boring nature, but for following the data where it lead . In this case it was for the conclusion that race was not the cause of some nations being poor. As Bryan Caplan notes, the original classical economists said the equivalent of “Black Lives Matter”, and for that were slapped with the label of “dismal”.

This new psychology would come into its own in defiance of many folk theories of how the mind works that so many people have built their lives around. I can imagine this would make a lot of folks very unhappy, but if Yarkoni is right — and I suspect he is — it may be a necessary sacrifice for psychology to make.

Losing the entertainment value of current research in favour of higher quality methodology, analysis, and communication may be unpopular, but is what’s needed to created a better map of the mind.