Seeing Like a Scientist
Or: Why ESP isn’t real, despite empirical evidence
In his 1998 book, Seeing like a state, James Scott focuses on the ways in which formal systems fail to capture the intricate object-level concerns of those tasked with executing the instructions prescribed by states:
Designed or planned social order is necessarily schematic; it always ignores essential features of any real, functioning social order. This truth is best illustrated in a work-to-rule strike, which turns on the fact that any production process depends on a host of informal practices and improvisations that could never be codified. By merely following the rules meticiously, the workforce can virtually halt production. In the same fashion, the simplified rules animating plans for, say, a city, a village or a collective farm were inadequate as a set of instructions for creating a functional social order, The formal scheme was parasitic on informal processes that, alone, it could not create or maintain. — Seeing Like a State, James Scott
No matter how comprehensive one’s instruction manual for creating a functioning state, there will always be hidden assumptions baked into the process — assumptions that must be adhered to, albeit unvoiced.
Reading about the literature on extrasensory perception (ESP) made me think about this problem of statehood and how it could be applied to science. To explain myself, I’ll first attempt to summarize the ESP research I’m referring to.
Extrasensory Perception: The Control Group for Science?
Scott Alexander has a good summary of what I’ll be talking about. To summarize his summary, there’s a few principles that, post-replication crisis, psychologists have converged on as best practice in research:
1. Demand very large sample size.
2. Demand replication, preferably exact replication, most preferably multiple exact replications.
3. Trust systematic reviews and meta-analyses rather than individual studies. Meta-analyses must prove homogeneity of the studies they analyze.
4. Use Bayesian rather than frequentist analysis, or even combine both techniques.
5. Stricter p-value criteria. It is far too easy to massage p-values to get less than 0.05. Also, make meta-analyses look for “p-hacking” by examining the distribution of p-values in the included studies.
6. Require pre-registration of trials.
7. Address publication bias by searching for unpublished trials, displaying funnel plots, and using statistics like “fail-safe N” to investigate the possibility of suppressed research.
8. Do heterogeneity analyses or at least observe and account for differences in the studies you analyze.
9. Demand randomized controlled trials. None of this “correlated even after we adjust for confounders” BS.
10. Stricter effect size criteria. It’s easy to get small effect sizes in anything.
Daryl Bem, a psychologist notable for believing in ESP had a study published in a prominent psychology journal purporting to prove ESP existed — that participants in his lab could “feel the future”. After his study was (correctly) criticized for methodological flaws, he comes back with a meta-analysis that follows the vast majority of the principles Alexander outlines, which Bem claims proves that ESP does exist.
There’s more to this story, but I’d recommend you just read Alexander’s original post to get a better picture.
Here’s where it gets really interesting: one theory is that there’s an experimenter effect — maybe who is doing the study changes the results? Two scientists, Wiseman (an ESP skeptic) and Schlitz (an ESP believer), looked at exactly that in a paper titled “EXPERIMENTER EFFECTS AND THE REMOTE DETECTION OF STARING”. In it, the experiments conducted by the ESP believer showed results in support of ESP, and those done by the skeptic showed the opposite. This happened despite both following the same methodology, with their opponent watching them the whole time. The explanation given was that subtle emotional cues influenced participant behaviour.
Implications for Scientific Research
Setting aside that this extended anecdote is a good argument for double-blinded methods, I think it shows something important about research: you can follow someone else’s methodology to the letter and get a different result — not only due to variation in subjects, but because of who you are as an experimenter.
Contrary to folk theories of science, research is not conducted by disembodied brains coldly evaluating objective evidence and reaching logically justified conclusions. Science is social in nature — there’s politics, conflicts of interest, pet theories, trust, and a host of other factors that influence the scientific process. Where James Scott said “we must never assume that local practice conforms with state theory”, I’d offer that with science we must never assume that practical methodology conforms with written methodology. No matter how precisely a paper outlines its methodology, it seems impossible for a replication attempt to entirely meet the conditions necessary for a “true” replication — only an approximate replication.
The good news is that the scientific community is developing social technology to deal with this problem. The many labs replication project has, as the name suggests, many labs working with common methodology to replicate old findings in the psychological literature. This avoids some of the problems of experimenter-caused bias in replications. While each lab may vary slightly in actual methodology, at least there’s a lot of them working with the same general instructions.
With Many Labs, we can start to see how any individual study may vary in its findings. If we only looked at the blue Xs — the original effect sizes — we may believe them to be very strong effects. When situated in the context of many replications, however, we can see that some of these findings may have simply been outliers where no real effect existed.
