See, Half Of 3-Sigma Detections Really Are False!!!

I've now written a couple blog posts on this topic and most specifically here where I discussed how CDF reported evidence for a new particle or force at the 3-sigma level.  Later, CDF reported the detection goes up to 4.8!!!-sigma after even more data is analyzed and even Sean Carroll at this point is musing that the result really might be true.

However D0, an independent collaboration looking for the same signal, has found that there is no new particle and that CDF was wrong and probably miss-calculated their background events.  They exclude CDF's result by 4-sigma and the plot above shows their data.  The red curve is the prediction of the standard model.  The black dotted curve is the supposed "CDF Bump" that could have been a new particle or something and the back error bars are what D0 found.

So people really should find some wisdom in the phrase "Out of the mouth of two or more witnesses" when throwing around things like 3-sigma and 4-sigma detections.  Why?  Because half of 3-sigma detections are false because the group doing the calculation often makes a mistake they didn't account for.  But the odds of an independent group making the exact same mistake in the exact same way and thus producing the exact same result is very minimal.

Now, I know what some of you are thinking: what if CDF did it right and D0 made the mistake?  Two things: first D0 got the expected answer and so there is a bit more confidence from that fact.  But secondly: this is why you need independent checks.  Somebody is making a mistake and unless independent checks are done you don't really know who it is.

The real golden rule for detections: It is often said that the golden rule for claiming detection is that 5-sigma == detection.  We now see how much a 4.8 sigma detection can be trusted.  I think the new rule for detection needs to be: 5-sigma out of the mouths of two or more independent witnesses == detection for the very reasons this latest result has shown.