How fundamental physics progresses
TL;DR: There is more than one way, and we are not exploring all of them
When discussing our approach to the foundations of physics, sometimes I bump into people that strongly object since “this is not how you make progress.” Some of them argue that new physical theories come when we are confronted with new data. The experiments always come first. Some of them argue that new physical theories come when trying to apply mathematics to physics in novel ways. Mathematics is always there before physics. Clearly, they can’t both be right. And it is suspect that the former tend to have an experimental physics background while the latter tend to be on the theoretical and philosophical side. Quite frankly, the attitude should be: whatever works. So, it’s useful to have in mind different historical examples to see how things can be different.
On the experimental side, there are clearly many cases in which the experimental data came first. Kepler’s laws were found by analyzing the experimental data collected by Tycho Brahe; quantum mechanics was clearly driven by an increasing set of experimental discrepancies from classical theory. So, yes, sometimes advances in fundamental physics are brought by analyzing experimental data. However, that is not always the case.
Anti-particles, and the positron in particular, were first hypothesized theoretically by essentially playing around with math. Dirac wanted to find a relativistic equation for the electron, found that 2x2 matrices wouldn’t work but 4x4 would. The use of group theory become more and more prominent, and the equations for electroweak unification and the strong force were effectively developed by trying different mathematical groups. The Higgs mechanism also came up as a mathematical trick, so much so that many flavors of that trick have been tried. This has given the impression to many following or working in high energy particle physics that new equations and ideas always come from trying to impose this or that mathematical structure, and see “what nature prefers.” But, again, things do not always work this way.
The full development of the equations for electromagnetism, for example, happened because Maxwell gathered the equations that were developed from experimental data and found that there was a mathematical inconsistency in Ampere’s law. In solving that inconsistency, that is in fixing the math at a conceptual level, he found the right equations. Those equations predicted electromagnetic waves. So, improvements in the foundations of physics can also come from cleaning up current theories.
Both special and general relativity, instead, were developed from physical principles. For general relativity, the basic idea was that gravitational and inertial forces are indistinguishable, as the elevator thought experiment illustrates. The consequence of this was that gravitation should bend light, because an observer in a non-inertial frame will see light traveling on non-straight paths. This effect was later found experimentally during a solar eclipse. In this case, improvements in the foundations of physics came from reasoning from first principles.
The point is that nobody knows how the new discoveries will come about. This is why we should be exploring all avenues, especially when the first option is not available: we do not have experimental data that breaks the Standard Model and likely won’t for a long time, if ever. However, most of the eggs in the foundations of physics are put in the second basket. Studying fundamental principles and assumptions of physics and cleaning up the current theories, what we do in our research program, is not something the funding environment promotes. This is probably the biggest problem in fundamental physics.