The evidence is that Dark Energy is responsible for the rate of expansion of the universe. The name makes it sound like a spooky force, but, in fact, it’s the cosmological constant, Λ, that Einstein added to his theory of gravity back in 1917. Einstein regretted it, but we have been forced to put Λ back in the theory to fit the evidence. There is theoretical backing of a sort from quantum theory, which also predicts the presence of a cosmological constant, Λ, but with a value that is far off what we need. Finding a way to unify the predicted value of Λ by quantum theory, and the observed value of Λ from the expanding universe, would be a great discovery. But even the most sophisticated theory is constrained by observational evidence, which always will be imperfect and incomplete, and hence our theories always will be an approximation, never an account of ultimate reality, argues James Peebles.
The standard theory for explaining the expansion of the universe from a hot dense state requires the presence of Einstein's cosmological constant. The usual symbol for it is the Greek Λ, and today Λ is more commonly known as Dark Energy. Maybe the name is an improvement, but nothing else has changed. Our well-tested theory of the expanding universe needs something that acts like Λ, but we still do not know how this Λ fits in with our standard physics.
SUGGESTED READING
Our model of the universe has been falsified
By Pavel Kroupa
It has been known since the 1930s that quantum physics naturally predicts the presence of a cosmological constant, Λ, but the predicted value is absurdly large. A quarter of a century ago, when we had much less evidence of how the universe is evolving, we tended to say that since the quantum estimate of Λ is clearly wrong, the only natural alternative is that Λ is zero. Einstein would have agreed with this hopeful argument; he regretted adding this term to his new theory of gravity, the general theory of relativity. But the accumulated evidence now makes a compelling case that we must learn to live with Λ, Dark Energy.
This is not the place to review the experimental case for the presence of Λ. My opinion of the evidence, and thoughts about how this evidence fits a physicist's philosophy of natural science, is in my book The Whole Truth [1]. You should bear in mind that the proper way to put the situation is that the experimental tests make a persuasive case that the standard theory of the expanding universe, including Λ, is a useful approximation to reality. The qualifier, approximation, is required because the final judgement of a theory lies with experimental tests. Most influential are tests of predictions that were not involved in the original formulation of the theory. We have abundant examples of this in natural science. And since all experiments have limited accuracy, all theories must be considered to be at best approximations to reality. Some are really successful, but they are approximations in my philosophy.
The best of our theories are wonderfully predictive. For example, consider the classical theory of electromagnetism that Maxwell put together a century and a half ago. This theory is used for the design of the energy transmission line from a power plant, which converts mechanical or solar energy into electromagnetic energy, carries it great distances, and distributes the energy to homes and industries. The same theory guided the electrician who installed the wiring in your home that brings in the electromagnetic energy that operates your refrigerator, powers your electric lights, and supplies the trickle of energy that allows operation of your cell phone.
___
You should be careful when you hear of an inconsistency between theory and observation about the nature of the expanding universe
___
Although Maxwell's theory is exceedingly useful it fails when applied to the properties of an atom. That is, our classical electromagnetism is an approximation; it is the limiting case of the theory of quantum electrodynamics. This better theory passes far more demanding tests. But quantum electrodynamics is, in turn, a limiting case of the quantum electroweak theory, which is part of the standard model for particle physics, which physicists feel surely could be put together better, if only we could see how.
I mean these remarks to show why you should be careful when you hear of an inconsistency between theory and observation about the nature of the expanding universe. The experimental tests are really difficult, and despite great care, errors can creep in. Wait until you hear about critical assessments and independent experiments that confirm or maybe contradict the evidence. If the inconsistency is confirmed, then bear in mind that although the theory is quite successful, it is not perfect. A real inconsistency points to where the theory needs improvement. We have no guarantee that the improvement that remedied the theory will be found, but it has happened many times before. Remember what happened with electromagnetism. The odds seem good that it will happen again, resulting in a better theory of Dark Energy, Λ.
When it comes to our theory of Dark Energy, the situation is made more complicated, and more interesting, by the fact that quantum physics predicts the presence of Λ. This goes as follows. An isolated molecule that has been cooled to its lowest energy state contains quantum mechanical zero-point energy. This energy is real; it must be taken into account to get the right quantum prediction of the energy required to pull the molecule apart. This was already understood by theory and established by experiment in the 1930s. It also was known that the same quantum theory predicts that electromagnetism has zero-point energy and that this energy behaves like the cosmological constant Λ, except that the predicted value of this quantum Λ is far too large [2]. As I said, our expanding universe theory is successful but approximate. Improving the approximation by explaining how the quantum prediction of Λ can be reconciled with the Λ of the expanding universe theory is a wonderful opportunity for a great advance in physics.
A popular thought is that the value of Λ changes as the universe expands. Maybe in the early stages of the expansion of the universe, the value of Λ was large, as quantum physics suggests it should be, and as required by the cosmological inflation picture of what the universe was doing before it started expanding. And perhaps as the universe expanded, the value of Λ decreased toward its only "natural" value, zero. So maybe Λ is small now because Λ has been decreasing for a long time, but it has not yet reached zero. This thought can be tested. The results so far are consistent with a constant value of Λ, but the tests can be improved. Convincing evidence of evolution would be a transformative discovery because it surely would give us a hint to how our physical theory could be adjusted to allow for a changing Λ.
Names can be useful for keeping track of ideas. Michael Turner invented the name for Λ, dark energy, in about 1998. Turner was at the University of Chicago. Bharat Ratra and I worked out the idea of an evolving Λ in the late 1980s. Ratra was with me at Princeton University in New Jersey at the time, and now is at Kansas State University. Our colleague Paul Steinhardt at Princeton gave the idea an elegant name, quintessence, and made important contributions to ideas about it. Quintessence is widely discussed; you might open your cell phone tomorrow, or I might open my newspaper, and read that some version of quintessence has been detected. As usual I advise you to suspend judgement of this interesting development until you hear about the checks and criticisms that would be sure to follow.
We must consider the other dark component of our standard theory, dark matter. I introduced it in 1982 to allow a more comfortable fit of the expanding universe theory to what we knew then. The idea was greeted with more enthusiasm than I considered warranted; it was only the simplest way I could see to resolve problems with the theory. I spent much of the 1990s devising other fixes to the theory. But the wisdom of the crowd was right, and by the year 2000 I felt pretty sure the dark matter picture is a useful approximation. Earlier, in 1984, I reintroduced the old idea of including the value of Λ. As before, it was meant to get the theory to continue to fit the growing evidence. The introduction of Λ was not so popular; it was so difficult to see how Λ could be reconciled with standard theory. But again I had pretty good evidence of dark matter then, the evidence was growing, and now we have a compelling case. So are dark matter and dark energy related? Here are things to consider.
___
Perhaps dark matter will never be detected, apart from its gravitational effects. Even so, that would not be an argument against its existence
___
Not long after the introduction of Dark Matter attempts began to detect the effects of possible interactions of dark matter with the baryonic (normal) matter we are made of. The experimental sensitivity has been greatly improved in successive generations of ever-better designs. It has not yet yielded credible evidence of detection. If you hear of an announcement of Dark Matter detection --- it could happen tomorrow --- then as usual I advise reserving judgement until you hear about reports of the friendly but intense search for errors. You might hear the argument that since greatly improved searches fail to detect Dark Matter, it seem likely to not be real. I don't agree. Natural science has had great success in studying the world around us, in examinations on scales small and large, and fitting the results to theory that passes tight tests of predictions, though always approximate of course. But we have no guarantee that we can continue this wonderful interaction of theory and observation. Perhaps Dark Matter will never be detected, apart from its gravitational effects. Even so, that would not be an argument against its existence. It would instead exemplify the obvious fact that we are limited in what we can hope to discover.
People have been debating ideas about how Dark Energy, Λ, and Dark Matter might be related. Maybe the unification of these entities could be extended to black holes. As we now know, they are not exactly dark, because when larger ones merge, they produce gravitational waves that can, and have been, detected. And Stephen Hawking gave us the wonderful prediction that black holes radiate energy, though at very low rates unless their masses are quite small. So, the current talk that black holes might be Dark Matter, or that black holes may have something to do with Dark Energy, is not implausible.
I mention all this because theorists are wonderfully inventive, and I expect that in time they will come up with a "final" theory that will take account of these ingredients in a logical and consistent way that agrees with all the experimental evidence that can be obtained by all the experiments the world economy can afford. Should that theory be accepted as reality? Or might it only mean that theoretical physicists can be wonderfully clever? I respect theoretical physics and have even tried my hand at it, but I respect even more the experimental evidence of what the world is like. That evidence always will be limited but it always will be essential. We do what we can with it, which has been a lot, and surely a lot more is to come. But in my philosophy, the world must show us how it operates, in successive approximations.
SUGGESTED READING
Questioning cosmic inflation
By Avi Loeb
A final thought. Natural scientists have made great progress by operating on the assumption that the world functions by rules we can discover, and each discovery is an improvement to the rules. I seldom hear it said, but the implicit assumption is that as theories about such things as Dark Energy improve, they are converging toward absolute reality. But we can never know whether we have converged to the final theory that would be the natural scientist's ultimate reality, or whether we only have a very good approximation. It has to be successive approximations, all the way down.
References
[1] P. J. E.Peebles 2022. The Whole Truth; a cosmologist's reflections on the search for objective reality. Princeton University Press
[2] Weinberg, S. 1989. The Cosmological Constant Problem. Reviews of Modern Physics 61: https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.61.1
Join the conversation