New Particle Or Force? And The Beauty Of Independent Experiments.

The latest buzz that we may have evidence of a new particle or force not predicted by any well established physics has reminded me of why I am glad there are several independent experiments looking at the same thing.  As I've discussed earlier, data analysis of complex data sets scares me, and I have learned enough doing my own data analysis that independent checks are crucial.

Tommaso Dorigo, who often has the best write ups of this type of thing, puts it like this:

It so happens that when experimental particle physicists search for something known, they bump into something they do not understand about their data. Most of the times, it is just a bug in their code, so physicists are accustomed to not grow excited in any way, but rather get a big cup of coffee and sit (possibly during nighttime) in front of the computer, painstakingly checking their code.. 

At that point, a more fun phase starts: the effect is studied systematically by using different simulations... 

Sometimes, however, the feature is hard to explain away with bugs or with insufficiently trustable simulations. At that point, a physicist has better start thinking in terms of what new physics model could be producing the effect he or she is seeing in the data. Another obvious thing that the physicist needs to do at this stage is to search for the same signal in other datasets which might likely be sensitive to it.... 

At the end of the lengthy process by means of which the physicists have tried to "kill" the signal they originally saw in the data, they will usually have grown confident enough to publish it.

And as implied above, data analysis is a lengthy problematic exercise that takes time before the person doing the analysis is confident that what he/she is seeing is real.  However, this stuff can be so difficult that even after such time has gone by and confidence is increased at the end of the day the age old maxim still seems to hold true:

Half of all three-sigma detections are false!

That unfortunately all too true rule of thumb is only realized in practice because  even after this painstaking process you still missed some systematic.  That is the worst!!!

However, if other independent groups get exactly the same answer, then you really can start being confident in the result because the odds are that two independent experiments aren't going to suffer from the exact same systematic presenting itself in identically the same way.

So, the Tevetron sees a 3-sigma "evidence" for new physics.  Hurray lets hope it's real!  But I will be even more happy if the two independent groups, CMS and ATLAS at the LHC, see the same thing.  If they do then I will really start believing the result is real.