Lecture 23: The Potential Outcomes Model

STATS 60 / STATS 160 / PSYCH 10

Concepts and Learning Goals:

Analyze the results of a randomized experiment using the potential outcomes model.

Review

• The best way to determine causality is to run a randomized experiment.

• Last class, we started a randomized experiment to examine whether retrieval practice causes better learning.

• Today, we will finish this experiment and analyze the results!

Quiz

Take 5 minutes to do this 4-question quiz.

Try your best, but don’t stress out—it doesn’t count towards your grade!

Answers

1. D

2. B

3. C

4. D

Please score your quiz (out of 4) and complete this poll.

If you don’t remember which group you were in, please ask!

Inference for a Randomized Experiment

Maybe the difference is just noise.

Even if the treatment had no effect, the difference wouldn’t be exactly zero.

How do we quantify the noise in a randomized experiment?

Potential Outcomes Model {.smaller}

Jerzy Neyman (1894-1981)

• In the potential outcomes model, each subject \(i\) has two “potential outcomes”.

  • \(Y_i(0)\) if they receive the control

  • \(Y_i(1)\) if they receive the treatment

• \(Y_i(1) - Y_i(0)\) represents the treatment effect for subject \(i\).

• Fundamental Problem of Causal Inference: We can only observe one of the potential outcomes, \(Y_i(1)\) or \(Y_i(0)\), because each subject is either assigned to treatment or control, not both.

Donald Rubin (1943-)

Potential Outcomes Table

It is easiest to visualize this model as a table.

$i$	$Y_i(0)$	$Y_i(1)$
$1$	$2$
$2$		$3$
$3$		$1$
$4$	$0$
...	...	...

Notice that we only observe one potential outcome per row.

The Null Hypothesis

The null hypothesis is that the treatment has no effect.

Under this null hypothesis, we can fill in the missing potential outcomes.

$i$	$Y_i(0)$	$Y_i(1)$
$1$	$2$	$2$
$2$	$3$	$3$
$3$	$1$	$1$
$4$	$0$	$0$
...	...	...

Randomness in a Randomized Experiment {.smaller}

The null hypothesis is that the treatment has no effect.

Under this null hypothesis, we can fill in the missing potential outcomes.

$i$	$Y_i(0)$	$Y_i(1)$
$1$	$2$	$2$
$2$	$3$	$3$
$3$	$1$	$1$
$4$	$0$	$0$
...	...	...

In a randomized experiment, the randomness is in the assignment of subjects to treatments.

Randomness in a Randomized Experiment {.smaller}

The null hypothesis is that the treatment has no effect.

Under this null hypothesis, we can fill in the missing potential outcomes.

$i$	$Y_i(0)$	$Y_i(1)$
$1$	$2$	$2$
$2$	$3$	$3$
$3$	$1$	$1$
$4$	$0$	$0$
...	...	...

In a randomized experiment, the randomness is in the assignment of subjects to treatments. (Perhaps subjects 1 and 2 were assigned to treatment instead of subjects 2 and 3.)

Depending on the treatment assignments, the difference in means will vary, even under the null hypothesis of no treatment effect!

Simulation in the Applet

Let’s set up the potential outcomes table in a spreadsheet.

Let’s copy these potential outcomes into an applet to simulate random assignments of subjects to treatment and control.

potential-outcomes.github.io{target=”_blank”}

Summary

Even if we run a randomized experiment, we still have to consider the possibility that the signal is just noise.

1. Every subject has a potential outcome under control, \(Y_i(0)\), and treatment, \(Y_i(1)\).

2. We can only observe one of these potential outcomes.

3. But under the null hypothesis that the treatment has no effect, we can fill in the missing potential outcomes.

4. We can simulate the different treatment effects we get under this null hypothesis to get a \(P\)-value.

Antonioli and Reveley (2005) investigated whether swimming with dolphins could be therapeutic for patients suffering from clinical depression.

• Recruited 30 subjects with a clinical diagnosis of mild to moderate depression.

• Randomly assigned subjects to one of two treatment groups of 15 subjects each. Both groups engaged in one hour of swimming and snorkeling each day, but

  • Dolphin Therapy group did so in the presence of bottlenose dolphins

  • Control group did not.

• At the end of two weeks, each subject’s level of depression was evaluated.

  • 10 subjects in the Dolphin Therapy group showed improvement

  • 3 subjects in the Control group showed improvement

Is this evidence that swimming with dolphins is therapeutic for patients suffering from clinical depression?

potential-outcomes.github.io{target=”_blank”}