That Alpha Error

In "On Downsizing the Status of Science," Dave Britton writes "... Truth is only what scientists can get each other to agree is true. And even then, on average one out of a hundred statistically significant findings of p < 0.01 is really just a coincidence."¹

Britton misinterprets the long-term average type 1 error rate (alpha, erroneously rejecting a true null hypothesis) for the frequency of statistical errors that occur in the published literature. On average, 1 of every 100 "significant findings" will be "just coincidence" only if one always tests only true null hypotheses with alpha = 0.01 established a priori. But in "real life," many of those results for which P < 0.01 will have occurred because the null hypothesis is false.

So, a result of "P < 0.01" that leads to a rejection of the null hypothesis may be a type 1 error, but it also...