Invisible Errors in Hypothesis Testing.
We've seen in this chapter and previously in Chapter 10 that a hypothesis test can lead to one of two results:
If the P-value is small (P-value < alpha) then it's unlikely that the difference we see between the null hypothesis value (parameter) and the observed sample value (statistic) is due to just sampling variablility so we reject the null and conclude that the evidence supports the alternative hypothesis.
If the P-value is too large (P-value > alpha) then we can't reject the null because it could be the numbers we're seeing in the sample are just due to sampling variability. In other words, we don't think the null is necessarily true but we don't have strong enough evidence against it; we haven't moved beyond "reasonable doubt".
So there's never 100% certainty from statistical analysis. Going back to Lance and the doping issue, we may have very strong evidence to reject his innocence and conclude that he's guilty, but there's still that tiny chance that he was telling the truth and his 4 positive test results were, in reality, false positives.
This shows that there's a tension between what statistical analysis reveals, based on sample data, and what is actually true in the real world population. A researcher can have solid data and do flawless analysis of the data and STILL end up making an (invisible) error.*
What we'll examine in this section is what those potential errors are called and what effect they can have on real world situations and decision-making.
*Note: This is why "metastudies" are so important! These studies review all of the research done on a particular research question and analyze the results.
References:
Notes:
Note: I have been unable to make a video with this notes due to the limitations of working on a laptop screen with a stylus (my writing just can't be detailed enough on the laptop to fill in the notes), so please look over the blank and filled-in notes, then see the videos and practice from Khan Academy listed below...they're great!
Videos: Section 8.3 Invisible Errors - Overview (page 1 of notes)
Practice problems (these are super useful, better than the questions in the text!)
Homework: Assignment #7 (8.3) on the Blue Sheet