Carl Sagan was wrong: ordinary evidence is enough

Do extraordinary claims require extraordinary evidence?

“Extraordinary claims require extraordinary evidence” said Carl Sagan; a dictum which asks us to use the scepticism of Hume to dismiss miracles and false claims. But what counts as extraordinary, and should common sense be our guide to it? An innate scepticism at the heart of science needs to be balanced and we should instead accept that the bizarre and counter-intuitive is what modern science is all about, argues Sean McMahon.

 

On the Ides of March in 44 BC, Julius Caesar was murdered on the steps of the Senate Chamber in Rome. As he collapsed amid the folds of his blood-soaked toga, his lungs released their final breath: about ten thousand billion billion molecules of nitrogen, oxygen, and so on. According to a popular and venerable factoid, with every passing minute you inhale a few of those same molecules. More than two-thousand years after they left Caesar’s lungs, they entered yours, just moments ago, wherever you are on Earth! That, surely, is an extraordinary claim. It seems enormously unlikely. But you can easily prove it on your own. With a little research, you will discover that the volume of Earth’s atmosphere is about ten thousand billion billion times the volume of a human breath (in or out). Once the molecules of Caesar’s last breath had spread and mixed evenly into the atmosphere, every breathful of air contained, therefore, about one of those molecules. A chemistry textbook will tell you that nitrogen hangs around for ages, so most of those molecules are still airborne: every minute you are bound to inhale (and exhale) some. QED. No extraordinary new evidence required.

20 10 12.lynch.ata SUGGESTED READING Why evidence won't change your convictions By Michael Lynch

Nevertheless, “extraordinary claims require extraordinary evidence” is a favourite saying of scientists, sceptics, and debunkers of all stripes. It was made famous by the astrobiologist and science communicator, Carl Sagan, who probably borrowed it from the sociologist Marcello Truzzi. Sagan and Truzzi moved in similar circles in the 1970s, when both were founding members of the Committee for the Scientific Investigation of Claims of the Paranormal. Truzzi and his fellow sceptics published sound advice about how extraordinary claims should be handled. Their arguments were careful and sophisticated, harking back two centuries to the great Scottish philosopher David Hume, who observed that “no testimony is sufficient to establish a miracle, unless the testimony be of such a kind, that its falsehood would be more miraculous than the fact which it endeavours to establish”.

Unfortunately, the famous dictum about extraordinary claims (sometimes known as Sagan’s dictum or Sagan’s standard) is often wielded in a rather less insightful way. It’s a good slogan: the repetition of “extraordinary” gives it a nice formal symmetry, which makes it seem almost self-evident, requiring no further thought or justification. But as we have just seen, at least some extraordinary claims do not require extraordinary evidence. We proved the claim about Caesar’s last breath with just a little textbook knowledge and a bit of arithmetic. You might object that this particular claim was not extraordinary in the right way for Sagan’s dictum validly to apply. But what is the right way? What does it take for a claim to be so “extraordinary” that it requires “extraordinary evidence” for us to take it seriously? What, indeed, is “extraordinary evidence”?

___

True scepticism aims to follow the evidence as diligently as possible, not simply to exclude radical or outlandish conclusions in advance

___

To think about this clearly we should start by recognising that “extraordinary” is a slippery word with multiple meanings. If we treat any proposition that seems out of the ordinary as an “extraordinary claim” then Sagan’s dictum becomes little more than a distillation of bias or prejudice; an excuse for not thinking harder. In that case, we would be better off without it. On the other hand, if by “extraordinary” we mean improbable, then Sagan’s dictum is certainly correct. We can use a bit of mathematics called Bayes’ Theorem to show that if a hypothesis is highly improbable to begin with then it requires extraordinary evidence to make it probable in the light of that new evidence. More precisely, the evidence has to be strongly predicted by the hypothesis, and at the same time so specific that it would be almost impossible to obtain if the hypothesis were false. To prove that you have won a lottery (perhaps a one-in-a-million chance?), you need to verify that you bought the winning ticket (similarly unlikely — but if true, then you really did win the lottery!).

Usually, however, neither hypotheses nor lines of evidence come with unequivocal probabilities attached to them. What is the probability that ghosts exist? It is not obvious how to answer this, but that doesn’t mean that Sagan’s dictum is useless here. To understand why, we can return to the writings of Marcello Truzzi. In an excellent 1978 article dealing with “claims alleging paranormal events”, he argued that “extraordinariness must be measured against theoretical expectations provided by the general body of scientific knowledge at the time.” In other words, claims require extraordinary evidence if they entail the falsehood of established scientific results that are themselves extensively tested and well understood. The existence of ghosts would be just such a claim, demanding revolutionary changes to the foundations of physics and biology. The evidence would need somehow to outweigh or account for all the physical and biological evidence supporting our existing expectation that ghosts are impossible. Of course, as the philosopher Karl Popper famously emphasized, a single counterexample (e.g., one black swan) is sufficient to refute a generalization (“all swans are white”). But there is a difference between a mere generalization and a well understood body of knowledge. A black swan is one thing; a swan that visits you from beyond the grave is something else.

20 09 18.berghofer.ata SUGGESTED READING Phenomenal evidence By Philipp Berghofer

Now we can see why some extraordinary claims require extraordinary evidence and some don’t. Those that do are either highly improbable (requiring highly unambiguous evidence) or counter to a weight of evidence that seems to rule them out (which the new evidence must somehow be reconciled with); those that don’t are simply out of the ordinary or counterintuitive, like the claim about Caesar’s last breath. Now, let us imagine a scenario in which scientists in Sagan’s own field, astrobiology, make a stupendous announcement: they have found a signal of extraterrestrial life. What happens next? I expect there will be a mass outbreak of Sagan’s dictum and wall-to-wall demands for extraordinary evidence (including from scientists themselves). But these demands may be unreasonable. We don’t know if life is rare or common in the universe, probable or improbable. There is very little evidence either way, so the claim isn’t really “extraordinary” in the sense required for Sagan’s dictum to apply. We might suspect that the scientists are wrong for technical reasons, or we might think they have neglected alternative explanations of their data because they are too excited by the prospect of a great discovery, but these worries should only motivate us to check that the normal burden of evidence has been met and to proportion our beliefs accordingly, not to impose a double standard by asking for something extra.

True scepticism aims to follow the evidence as diligently as possible, not simply to exclude radical or outlandish conclusions in advance (Truzzi called that “pseudoskepticism”). Sagan knew this better than anyone, arguing that science must balance “the most ruthless skeptical scrutiny” with “an openness to new ideas, no matter how bizarre or counterintuitive they may be”. Indeed, the more we learn about the universe, the more bizarre, counterintuitive, and extraordinary it seems to be.

______________________________________________________________________

The ideas in this article are elaborated in more detail in: McMahon, S. (2020) Do extraordinary claims require extraordinary evidence? The proper role of Sagan’s dictum in astrobiology. In: Smith, K.C., & Mariscal, C. (Eds.), Social and Conceptual Issues in Astrobiology. Oxford University Press, New York.

Latest Releases
Join the conversation

Scott Anderson 14 January 2023

Interesting concept considering all of modern physics relies on "inference" which is NOT evidence, but a form of confirmation bias.

The math has holes, known as "ideal-states" imaginary starting positions with no grounding in reality. Time has never been observed to stop for an "inertial frame", the atoms have not ceased motion for absolute zero, and we have never reduced the mass of a vessel to zero, the perfect vacuum has never been achieved all attempts structurally fail.

Zeros in the math, when a zero has never been observed.

Ideal states are not useful. They poison the well. Seriously intelligent people blindly following the math to outrageous conclusions, looking for wormholes, timetravel, and other unobserved nonsense. It's been over a century, modern physics has failed at producing a congruent system top to bottom. Dark matter/energy represent the difference between prediction and observation, instead of starting over we doubled down folding them into the problem. Remove the fantasy, remove the ideal states. The answer is simpler than it seems but we have to get out of our own way first.... too far down the line too much momentum and pride in what we assume to know. Without a proper place for gravity we cannot claim to know anything for certain, everything is slightly incorrect. There is a significant difference between quantitatively accounting for the downward acceleration, and knowing what gravity is. Sure as hell isn't curved spacetime. If Einstein or Newton understood gravity we wouldn't need dark matter to make up the difference in galactic behavior, we must start over.