Worksheet 20: Multiple testing#
Your name:
Your student ID number:
Suppose our data really is just random noise (the null hypothesis is true), but we perform the same experiment \(100\) times.
If we set our level for rejecting the null hypothesis at \(\alpha = 0.05 = \frac{1}{20}\), how many times would we expect to end up rejecting the null hypothesis?
Is the \(p\)-value a random quantity? Why?
If you increase the level, will the fraction of false positives increase or decrease?
In scientific publication, it is standard to require the level \(\alpha = 0.05\). What percent of published statistically significant trends do you expect are actually random noise?
Why shouldn’t I just set my level \(\alpha = 1/100000\) or even \(\alpha = 0\)? Then I will never get a false positive.
Can you think of any situations where you would naturally want to do a lot of different experiments in parallel?
Are you psychic? How many cards did you guess correctly? What was your \(p\)-value?
Do you have ESP? Formulate a hypothesis test.
a. What is your null hypothesis?
b. What is the \(p\)-value (in plain English)? How would you compute it?
c. The ESP applet gave you a \(p\)-value. Do you reject the null hypothesis?
What is the chance that, as a class of \(n = 50\), we get at least one false positive?