The Perpetual Question -- Theory vs Experiment

The last couple of posts have brought up a couple questions that I'd like to throw out there to get some commentary on. How does science move forward? Specifically, what is the relationship between theory and experiment in the advancement of science? In just the last couple posts, we have looked at two ideas about nature. I will add to the list some examples including Flat Earth ideas, the experience of Fresnel, Poisson and Argo in discovering the "Poisson Spot", the experience of Alfred Wegener related to continental drift, Galileo promoting Copernican Cosmology, and "Brane Cosmology". All these ideas have been initially dismissed outright as false or at least unscientific. All appeared to be so at the outset. Under further investigation, some showed no scientific merit, some were shown to be correct and only later was a theoretical explanation discovered, and on some the jury is still out.

• In the case of the Poisson Spot, Fresnel's theory of light gave an unbelievable, but testable, prediction. The prediction was tested experimentally and was shown to be correct. The theory led the experiment.


• In the case of Continental Drift, Wegener's observations required mechanisms that seemed impossible. It was only decades later that his observations were vindicated, a theoretical explanation was given, and he was shown to be correct. The observations led the theory.


• In the case of the Flat Earth ideas, observations by various people made predictions that appear impossible. The hypothesis even gives testable predictions that appear to be verified by experiment! The ideas are still not accepted (obviously). (I have never actually done the experiment that I just referenced in the link. I'll take the guy's word for it, because it would be all-too-easy to check his experiment and refute it, so I doubt he'd give an easily falsifiable prediction. It is actually non-trivial to show why the referenced experiment is not valid. If you have a good explanation, give it in the comments.) The theory (along with a bunch of other experiments and observations) dismissed the experiment.


• In the case of Brane Cosmology, and even String Theory or the Holographic Principle, the theories are mathematically beautiful and explain many things, but make few (if any) testable and falsifiable predictions. (I'll avoid too much discussion here, because I am stepping way outside my area of expertise among others who are vastly more familiar with the details of the theories.) Their acceptance among the scientific community is mixed due to lack of experimental evidence. The experimental uncertainty (partially) dismissed the theory. The jury is still out.


• In the case of things like the Mpemba effect, the experimental results are mixed and the theoretical foundation is ambiguous. It does not really challenge modern scientific theory, just common sense. Its acceptance is mixed. (Mostly it is ignored.) The jury is still out.

My question is, in your view, how does/should science confidently accept or dismiss an idea about the world? Many ideas that contradict then-modern scientific understanding, or at least common sense, are at first discarded, but later shown to be correct. Many other ideas that contradict scientific understanding are discarded despite what may appear to be sound experimental verification, and then only later were the experiments shown to be invalid or improperly interpreted. Still other ideas are established based on a few experiments or theoretical predictions, and immediately accepted. There are examples of this type that have met both fates -- continued acceptance or eventual falsification.

I just want to get some views on how does/should this all play out. The scientific method is rigorous, but what other factors come into play? Comments.