The 3 key steps to overthrowing a scientific theory

Science, like many things in life, is always a work-in-progress. While a successful scientific theory has questions it can answer, natural phenomena it can accurately describe, and robust predictions it can make, it’s also fundamentally limited at any point in time. Any theory, no matter how successful, has a finite range of validity. Stay within that range and your theory works very well to describe reality; go outside of it, and its predictions no longer match observations or experiments. This is true for any theory you pick. Newtonian mechanics breaks down at small (quantum) scales and high (relativistic) speeds; Einstein’s General Relativity breaks down at a singularity; Darwin’s evolution breaks down at the origin of life.

But the models we have today are known to only be our best approximations of reality. Somewhere, if we look hard enough, we’ll find where the limits of even our best theories lie, and find evidence that contradicts the prevailing theory’s best predictions. When we come upon a better theory that gets it right, where we can hold that new theory up against our old theory and can compare their predictions, one against the other, and the new theory scores an overwhelming victory, that’s usually the precipitating event for a scientific revolution. It’s happened before, and we can be certain it’ll happen again.

One of the great puzzles of the 1500s was how planets moved in an apparently retrograde fashion. This could either be explained through Ptolemy’s geocentric model (left), or Copernicus’ heliocentric one (right). However, getting the details right to arbitrary precision was something neither one could do. Both models have little predictive power; they could not detail the orbital properties of a hypothetical additional planet.

The prerequisite

If you hope to supersede the current, best theory we have, and this is true in any scientific discipline, you have to honestly recognize the present successes, failures, and limitations of your theory. Every theory that we have makes predictions, but there’s a limit to what they can successfully predict. So long as that’s the case, there will always be the option for a better, more complete, more fundamental theory to help us understand the Universe.

The proverbial holy grail of scientific theories is what’s called a final theory of everything. This was Einstein’s ultimate dream, and remains the dream of many other scientists across a variety of fields. Such a theory would predict all natural phenomena in the Universe given any initial setup and conditions. You could calculate the outcome of any experimental setup in advance; you could predict how any system would evolve arbitrarily far into the future. The only limitation you’d face would be from not having an arbitrary amount of computational power, rather than any theoretical limitations.

The Standard Model particles and their supersymmetric counterparts. Slightly under 50% of these particles have been discovered, and just over 50% have never shown a trace that they exist. Supersymmetry is an idea that hopes to improve on the Standard Model, but it has yet to achieve the all-important ‘step 3’ in the three steps for supplanting the prevailing scientific theory.

But let’s be honest about what we know and do not know today: we have not arrived at that point. We do not have a working theory of everything; we have a slew of very successful theories, each one of which is fundamentally limited in its scope. In every field, we have phenomena we can observe or experiments we can design where the predictions of our best theory either contradict the data or yield nonsense. In addition, there are often problems or puzzles that cannot be explained with the theories we have.

• Why do neutrinos have mass?
• Why does the Universe consist of large amounts of matter but not antimatter?
• What happens to the gravitational field of an electron as it passes through a double slit?
• And why do the fundamental constants have the values that they have?

An unexplained phenomenon that is observed, but doesn’t have a theory to predict it, is often the impetus for a scientific revolution. This has to be the default starting point: the point at which all scientists in all disciplines can come to the table and say, “Yes, I agree that this is where we begin our investigations.”

In Newton’s theory of gravity, orbits make perfect ellipses when they occur around single, large masses. However, in General Relativity, there is an additional precession effect due to both the curvature of spacetime and the fact that the planets are in motion with respect to the Sun, and this causes the orbit to shift over time, in a fashion that is sometimes measurable. Mercury exhibits the largest such effect within our Solar System, precessing at a rate of an extra 43″ (where 1″ is 1/3600th of one degree) per century due to this additional effect.

Step 1: Your new theory must reproduce all the successes of the leading theory.

So, you’ve got a new theory that you hope will supplant the presently leading one? Great! Your first order of business is to demonstrate that your new theory doesn’t fail where the old one succeeded. The more successful the prevailing theory, the taller an order it is to meet this goal. For example, if you wanted to supersede our current theory of gravity: Einstein’s General Relativity? You’ve got to explain:

• gravitational lensing,
• the precession of Mercury’s orbit,
• the Lense-Thirring effect,
• gravitational redshift,
• the decay of binary pulsar orbits,
• the Shapiro time delay,
• and gravitational wave signals from merging black holes and neutron stars,

along with many other experimentally verified realms where General Relativity has been validated. From tabletop experiments on Earth to Solar System-scale experiments to Universe-scale measurements of galaxies, galaxy clusters, and the cosmic web, your new theory has to do at least this well: succeed where the leading theory has already succeeded.

Numerical simulations of the gravitational waves emitted by the inspiral and merger of two black holes. The colored contours around each black hole represent the amplitude of the gravitational radiation; the blue lines represent the orbits of the black holes and the green arrows represent their spins. The act of accelerating one mass through a region of curved spacetime will always lead to the emission of gravitational waves.

The same thing is true if you wanted to go beyond Darwinian evolution. Yes, Darwin’s theory has its limitations, but it does an exquisite job of explaining a great number of observed phenomena, including:

• the emergence of biological diversity,
• the responses of organisms to various selection pressures,
• and the inheritance of traits among child organisms from their parent organisms,

as well as many others. Your first step, if you want to supersede Darwinian evolution, is for your theory to also achieve equally satisfactory explanations for all of these.

Similarly, if you were determined to improve on the Bohr model of the atom, you’d have to reproduce its successes, including:

• explaining the various energy levels within an atom,
• explaining the scattering experiments that show the presence of an atomic nucleus,
• and the spin-orbit interaction of the electron with the atomic nucleus,

among others. These feats are not necessarily easy to achieve. Additionally, this also means your new theory cannot make new predictions that contradict observations that have already been made or experiments that have already been performed. It isn’t enough to get a selection of these predictions right; you have to reproduce every single success of the prior theory. If you can’t equal what you’re trying to replace, you won’t surpass it.

A light-clock, formed by a photon bouncing between two mirrors, will define time for any observer. Although the two observers may not agree with one another on how much time is passing, they will agree on the laws of physics and on the constants of the Universe, such as the speed of light. A stationary observer will see time pass normally, but an observer moving rapidly through space will have their clock run slower relative to the stationary observer.

Step 2: Your new theory must succeed where the prior theory does not.

It’s true that theorists play in the sandbox all the time: even if our current explanation is 100% satisfactory, we’re always looking for new solutions to old problems. This kind of tinkering can often pave the way for a scientific advance down the road, where a critical point is reached: where there’s some sort of conflict or gap between what we theoretically expect to see and what we’re actually observing.

We’re only ever able to discern whether one theory is better than another theory, however, by contrasting them against one another and testing which one better matches what’s occurring in the real world. This usually requires a motivation; for someone to look at the data and notice that our predictions weren’t matching up with reality.

When this occurs, it’s natural to suspect that something isn’t right with the old theory; there’s something it simply fails to explain. Newtonian physics couldn’t explain the mechanics of fast moving particles; the ray theory of light couldn’t explain interference patterns; the universal law of gravitation couldn’t account for Mercury’s orbit.

The orbits of the planets in the inner Solar System aren’t exactly circular, but they’re quite close, with Mercury and Mars having the biggest departures and the greatest ellipticities. In the mid-19th century, scientists began noticing departures in the motion of Mercury from the predictions of Newtonian gravity: the differences were small, but measurements were so accurate that the discrepancy couldn’t be ignored.

All of these puzzles led to many new ideas which would explain these phenomena, but not every idea could also reproduce the pre-existing successes. For example:

• A hypothetical planet interior to Mercury — dubbed Vulcan — was proposed by Urbain Le Verrier to explain Mercury’s anomalous orbit.
• Other scientists proposed that the Sun’s corona was massive, and impacting the motion of Mercury.
• Another team, Simon Newcomb and Asaph Hall, determined that if you replaced Newton’s inverse square law, which says that gravity falls off as one over the distance to the power of 2, with a law that says gravity falls off as one over the distance to the power of 2.0000001612, you could explain Mercury’s motion.
• Finally, Einstein did away with Newton altogether, replacing his gravitational “action-at-a-distance” with curved spacetime.

All of these ideas were seriously considered for many years; this is part of why steps 1 and 2 aren’t enough on their own. If you want to establish which of the theoretically possible explanations actually best reflect the real universe that you inhabit, you have to confront your ideas with the all-important third step.

The hypothetical location of the planet Vulcan, presumed to be responsible for the observed precession of Mercury in the 1800s. Exhaustive searches were performed for a planet that could have accounted for the anomalous motions of Mercury in the context of Newtonian gravity, but no such planet exists, falsifying the prediction of an interior planet in our Solar System.

Step 3: Your new theory must make new, testable predictions that differ from those of the original theory.

Each one of these ideas for a new theory of gravity or massive phenomenon in the inner Solar System would have observable consequences: consequences that would distinguish them from not only one another, but from the prevailing, older theory of Newtonian gravity.

• If there were a new planet interior to Mercury, it should have been detectable with a telescope.
• If the Sun’s corona were massive, we should detect a greater particle/matter density than would be consistent with observations.
• If Newcomb & Hall’s theory of gravity were correct, it would affect the observed orbits of the Moon, Venus, and Earth in ways that do not match with observations.
• And if Einstein were correct, it would have meant that, with space being curved by mass, that a background light source should follow a curved, rather than a straight, path.

In fact, according to Einstein’s General Relativity, it became possible to calculate, for any mass at all, exactly how severely a light-path would be curved-and-bent. There’s one very large mass in our Solar System, in fact: the Sun. If Einstein’s predictions were correct, a total solar eclipse could prove to be the perfect time to test it. How much would a background point-of-light, like a star, be bent by the Sun’s gravity?

• Would it curve according to the amount General Relativity predicted?
• Would it curve by a null amount, since light has no mass and should experience no Newtonian attraction?
• Would it curve by the amount you’d get in Newtonian gravity if you assigned a photon a mass-equivalent given by its energy: through Einstein’s E = mc²?

The results of the 1919 Eddington eclipse expedition showed, conclusively, that the General Theory of Relativity described the bending of starlight around massive objects, overthrowing the Newtonian picture. This was the first observational confirmation of Einstein’s theory of gravity.

In 1919, during a total solar eclipse, this prediction of Einstein’s was put to the critical test. In very short order, the results were published and looked extremely strong: light bent according to Einstein’s predictions, and definitively not according to the predictions of any of the alternatives that had been put forth. In a tremendous revolution, we had a new theory of gravity, put to the test many times over the past 104 years, and passing that test wherever the observations or experiments were of high-enough quality.

Travel the Universe with astrophysicist Ethan Siegel. Subscribers will get the newsletter every Saturday. All aboard!

It took similar theoretical developments and experimental/observational confirmation to arrive at all of our leading scientific theories, from genetics and DNA to the Big Bang, cosmological inflation and dark matter. These aren’t our greatest theories because the math is so pretty or they match our intuition so well, but because they describe the natural phenomena that we actually observe in this Universe so successfully.

This four-panel animation shows the individual galaxies present within Abell 2744, Pandora’s Cluster, alongside the X-ray data from Chandra and the lensing map constructed from gravitational lensing data. The mismatch between the X-rays and the lensing map, as shown across a wide variety of X-ray emitting galaxy clusters, is one of the strongest indicators favoring the presence of dark matter. Lensing, importantly, is another explicit but imaginative prediction of General Relativity that was recognized “must exist” long before it was ever observed.

As science becomes a more developed, evidence-rich enterprise, it becomes a more herculean task to create a single theory that explains the full suite of data. Yet that’s exactly what the most successful theories do: explain so much of that data in such great detail over an extremely large range of validity. However, we must always keep this in mind: no matter how successful an idea has been in the past, all it takes is one inconsistent observation to throw the whole thing into doubt. Our greatest scientific theories of today will most likely all be superseded someday in the future, particularly as new and superior evidence is gathered.

Massive neutrinos are a hint of physics beyond the Standard Model; the black hole information paradox is a hint of gravity beyond General Relativity; the facts that atom-based matter exists and that life arose from it are undeniable, but where that matter (and not an equal amount of antimatter) came from and how life arose from it are still both great unknowns. These puzzles and paradoxes, along with many others, may serve as harbingers of an eventual, perhaps even monumental, scientific advance. Until then, we can only speculate at the frontiers of science, in our attempts to take these three massive steps toward a better understanding of the Universe.