The way we do science is broken: frequently inefficient, unreliable and even totally invalid, the road to consensus is fraught with hostility and academic pettiness. Cory Clark and Philip Tetlock argue for an alternative to the critique-reply-rejoinder format with the radical potential to transform the pursuit of knowledge.

If we were naïve observers, we might think of scientists as earnest detectives—carefully sifting through the evidence, pursuing all reasonable leads, and updating their beliefs as needed. We might imagine that scientists get together, exchange notes, form brilliant and empirically accurate beliefs, and then share these state-of-the-science ideas with the public. These ideas usually would be true and thus form a reliable basis for designing effective interventions and policies.

To be sure, science has accomplished remarkable feats, from vaccines to spacecrafts. But science is far from the idealistic version portrayed above. Science is the single most effective mode of knowledge formation to date. But, it can also be inefficient, hostile, petty, unreliable, and invalid. An average disagreement in science looks a bit like this: Scholar A forwards an idea. Scholar B contradicts it. Scholar A mocks Scholar B in a commentary, collects more data, and finds herself correct. Scholar B mocks Scholar A in a commentary, collects more data, and finds himself correct. Over time, they come to hate one another, and through the collection of more and more data, become only more convinced of their own initial beliefs. In good cases, third-party observers eventually make sense of the mess of seemingly contradictory evidence, and a “scientific consensus” is born.

___

Intellectual adversaries treat one another as enemies in a turf war, and so they strive to eliminate one another.

___

Scientific Indignities

Over the past decade or so, many scholars have accepted that much of science suffers a “replication crisis”. When a group of scholars tries to conduct the exact same methodological procedures as an earlier set of scholars, they often find different (and usually much less impressive) results. This means that a great deal of science is unreliable—very similar studies do not consistently produce very similar results.

But things are a bit worse than that. A great deal of science is also invalid, meaning it is simply not true. Even highly replicable findings can be wildly misleading, such as when a highly replicable association between two variables (say, ice cream sales and shark attacks) is accompanied by a highly inaccurate causal story (purchasing ice cream causes sharks to attack). Scholars had to work very hard to detect and demonstrate the replication crisis. The validity crisis is much simpler to detect: There are countless contradictory claims in the published literature.

STEM fields are biased against women. STEM fields are biased against men.

Stereotypes are generally false. Stereotypes are generally true.

Psychological sex differences are caused by evolution. Psychological sex differences are caused by culture. Psychological sex differences mostly do not exist. Biological sex isn’t really a thing.

Such claims, at least taken at face value, cannot all be true. Either someone is horrifically wrong, or at least someone is exaggerating. Although science purports to pursue truth, science actually incentivizes such contradictions.

  SUGGESTED READING Quantum mechanics gives us power, but no answers By John Horgan

Be New

Prestigious academic journals and institutions incentivize scholars to forward new ideas and findings—ideas that distinguish them from earlier scholars and contribute something unique to human knowledge. In other words, scholars are incentivized to create the impression of disagreement, even if disagreements are nominal, minuscule, or even non-existent.