Myths of rigor and illusions of discovery are holding science back. It’s time for an overhaul, writes Simine Vazire.
Scientists have a problem. We are discovery junkies. The addiction metaphor is overused, but, well, I can’t resist. Scientists have a drive to discover the next big thing. This drive can be channeled in positive ways, but can do serious damage to science and to society when it goes unchecked. And what’s worse, the journals we publish our science in are the enablers that pretend they are protecting us.
The problem runs deep. Science selects for people who are naturally curious, and then it heaps rewards and incentives that amplify their drive to find new things. Unfortunately, not all findings are equally gratifying, or equally valued – learning that your potential new cure (for cancer, or racism, or inequality) doesn’t work is just as informative as finding out that it does work, but it just doesn’t feel nearly as good. What’s more, the negative answers aren’t rewarded nearly as much as the positive ones. Journals don’t want to publish the negative results. Awards are rarely given out for rigorously testing a good idea that turned out to be wrong. A track record of negative results is not going to get you a grant.
A system that only publishes positive results and sweeps negative results under the rug would be bad enough, but it gets worse. There’s good reason to think that in some fields, many of the positive results aren’t real discoveries – they are quite likely to be false positives. In my field of psychology, the evidence of this problem is piling up, even if many of the leaders in the field still deny it. When we try to replicate influential studies that have been considered sound enough to put in textbooks, we often find that we can’t get the original finding. When we change the research process to put safeguards in place to reduce bias (for example, by getting authors to commit to their research plan in advance, using a mechanism called “Registered Reports”), we find that about half of the discoveries disappear. Even worse, when we simply try to reproduce the original study’s results using the original study’s own data, we often can’t do it. We know false discoveries are a big problem in psychology because we’ve looked. Most fields haven’t even begun to look.
We know false discoveries are a big problem in psychology because we’ve looked. Most fields haven’t even begun to look.
This seems like something that should worry scientists – if not the scientists making the false discoveries, at least the journals and editors that are being duped and publishing them. Unfortunately, in many sciences, no one in the system is particularly motivated to find out if a discovery is real or not. We all enjoy the warm glow of what feels like a discovery, and we all lose if we dig too deep and realize the discoveries aren’t real. So it’s not exactly that journals and editors are being duped – in fact, they’re a big part of the problem.
Issue 94, 1st February 2021
Join the conversation