The logic of NHST is easily stated. We compare our actual sample with the expectation of the null hypothesis.
- We know the sampling distribution for the null population: the samples we could expect.
- We choose a portion of it that corresponds to 5% of the total distribution.
- The probability of the null population producing a hypothetical sample that lies in that portion is low (5%).
- If the actual sample does lie in that area, then we say that we reject the null hypothesis because the sample and the expectation of the null hypothesis are inconsistent.
Normally, of course, the portion chosen is one or both tails of the sampling distribution. This allows us to describe the portion as the set of samples that are too different from the null population to have come from it.
However, the strict logic works equally well for any other portion of the distribution, such as one centred over the mean of the sampling distribution (which is zero). This version would allow us to describe the portion as the set of samples that are too similar to the null population to have come from it. (Fisher himself suggested this)
Why do we accept the first use of the NHST logic, but not the second? The answer is that the first corresponds to our intuition that there will be another hypothesis/population that will produce our population more often than the null population. Indeed, we can only reject the null hypothesis on the basis that there is an alternative: an implied second hypothesis (usually called the alternative hypothesis).
That 4th step in the logic of rejecting the null hypothesis means that we are comparing two hypotheses. We cannot do that without a prior distribution for the two hypotheses – and we probably don’t have one because the alternative hypothesis isn’t even defined.