Probability & Statistics · Lecture 5 of 5

Causal inference

Which comparison answers the question “what if we intervene?”

1 Effects and selection2 Designs3 Adjustment

Developed from Ian Crawford’s lecture material. Notes, slides and PDF: fatih.ai/probabilityandstatistics

Opening question

The treated score 16.67 points lower. Did the treatment hurt them?

Seven people, outcomes in invented points. People 1, 3 and 4 were treated.

Person	1	2	3	4	5	6	7
Treated?	yes	no	yes	yes	no	no	no
Outcome	30	30	46	54	60	70	80	treated mean $43.33\text{,}$ others $60$

Your answerVote now. Part 1 answers it.

AYes, by 16.67

BNo

CCannot tell yet

Why start here

Programme evaluations, university premiums and pay gaps all start from a comparison like this one. The question is what else differs between the groups.

Lecture 5 · The route

Define the effect, then find a comparison that identifies it

Part 1

The effect we want and the gap we see

What is a causal effect, and why does a difference in group averages not measure it?

potential outcomesATEATTselection bias

Notes 5.1–5.2

Part 2

Designs that remove selection

How can the way treatment is assigned make selection bias zero, and for whom is the answer then valid?

random assignmentITTcompliersLATEexternal validity

Notes 5.3, 5.4, 5.7

Part 3

Adjusting for covariates

Without randomisation, when does comparing like with like recover the effect, and when does “controlling for” make things worse?

CIAoverlaptarget weightsbad control

Notes 5.5–5.6

By the end of this lecture you can

Define potential outcomes, ATE and ATT, and explain why one potential outcome per person is always missing.
Derive $\Delta=\mathrm{ATT}+B$ and argue the likely sign of selection bias in an economic comparison.
Explain what random assignment identifies, and compute ITT and the Wald ratio for compliers.
Judge an adjusted comparison: CIA, overlap, the right weights, and why occupation is a bad control for pay.

Part 1 of 3

The effect we want and the gap we see

What is a causal effect, and why does a difference in group averages not measure it?

Key terms

potential outcomesATEATTselection bias

Notes 5.1–5.2 ↗

Why it matters

People choose programmes, universities and jobs partly on their prospects, so participants differ from non-participants before any treatment. A naive comparison could close a helpful programme, or credit a university with earnings its students would have had anyway.

In this part

5.1Potential outcomes: one person, two outcomes, one observed
5.1ATE and ATT: whose effect?
5.2Every observed gap is ATT plus selection bias
▶Animation: how selection bias can hide a positive effect
5.2Selection can push a comparison either way

5.1Two worlds for one person

Each person has two potential outcomes; we only ever see one

Definition

Potential outcomes

For unit $i$ and treatment $D_i\in\{0,1\}\text{:}$ $Y_i(1)$ is the outcome under treatment, $Y_i(0)$ without it. The individual effect is $\tau_i=Y_i(1)-Y_i(0)$ . The outcome not realised is the counterfactual.

Key formula Identity (1)

The observed outcome

Y_i=D_iY_i(1)+(1-D_i)Y_i(0)

In words we observe $Y_i(1)$ if treated and $Y_i(0)$ if not, never both.

Needs a well-defined treatment, and no effect of one person’s treatment on another’s outcome.

One person, two possible outcomes, one observation

So $\tau_i$ is never observed for anyone: causal inference is a missing-data problem.

5.1Average effects

ATE and ATT average the same effects over different people

Key formula Definition (2)

Average treatment effects

\begin{aligned}\mathrm{ATE}&=E[Y(1)-Y(0)],\\ \mathrm{ATT}&=E[Y(1)-Y(0)\mid D=1]\end{aligned}

In words the average effect over everyone, and over the treated. Both terms of an effect average over the same people.

ATE and ATT differ when effects vary and the treated are not typical: $\mathrm{ATE}=P(D=1)\,\mathrm{ATT}+P(D=0)\,\mathrm{ATU}\text{.}$

The notes’ toy population (invented points); bold = observed
Person	$D$	$Y(0)$	$Y(1)$	$\tau$
1	1	20	30	10
2	0	30	32	2
3	1	40	46	6
4	1	50	54	4
5	0	60	64	4
6	0	70	80	10
7	0	80	78	−2

$\mathrm{ATE}=34/7\approx4.86\text{;}$ $\mathrm{ATT}=(10+6+4)/3\approx6.67\text{.}$

ATT subtracts the treated people’s own no-treatment outcomes, not the control group’s observed mean.

5.2An identity, not an assumption

Every observed gap is the effect plus selection bias

Key formula Identity (3)

Observed difference = effect + selection

\begin{aligned}\Delta&=E[Y\mid D=1]-E[Y\mid D=0]\\&=\mathrm{ATT}+B\end{aligned}

In words add and subtract $E[Y(0)\mid D=1]\text{,}$ the treated group’s unseen no-treatment mean.

Definition

Selection bias

$B=E[Y(0)\mid D=1]-E[Y(0)\mid D=0]\text{:}$ how the treated would have fared untreated, compared with the untreated.

Worked example the toy population

The treated would have averaged $110/3$ untreated; the controls average $60\text{.}$

\begin{aligned}B&=110/3-60=-70/3\approx-23.33\\ \Delta&=20/3-70/3=-50/3\approx-16.67\end{aligned}

Common mistake

Selection bias is not sampling bias. It arises even when the whole population is observed.

Identification is whatever justifies learning the unseen $E[Y(0)\mid D=1]\text{.}$

Final frame: seven people on one outcome axis. Treated people 1, 3 and 4 have observed outcomes 30, 46 and 54 and revealed no-treatment outcomes 20, 40 and 50, joined by green arrows of +10, +6 and +4; control people 2, 5, 6 and 7 have observed outcomes 30, 60, 70 and 80, with their treated outcomes unknown. Group means: treated observed 43.33, treated without treatment 36.67 (dashed), control 60. Below the axis, three arrows: Δ ≈ −16.67 from 60 to 43.33; B ≈ −23.33 from 60 to 36.67; ATT ≈ +6.67 from 36.67 to 43.33. Beside it: observed treated 130/3 ≈ 43.33, control 60; Δ = 130/3 − 60 = −50/3 ≈ −16.67; revealed Y(0) = 20, 40, 50 for the treated; ATT = (10 + 6 + 4)/3 = 20/3 ≈ 6.67; Δ = ATT + B, −16.67 = 6.67 + (−23.33); B = 110/3 − 60, lower baselines, not harm. A negative comparison can hide a positive effect; recovering ATT needs evidence about B.

5.2Reading the sign of B

Selection can push a comparison either way

Worked example a university premium (illustration)

Graduates of university O earn $£4{,}000$ more than those of X: $\Delta=\mathrm{ATT}+B\text{.}$ If O’s students would have earned more even at X, then $B>0$ and $\mathrm{ATT}<£4{,}000\text{;}$ $B=£3{,}000$ would leave $£1{,}000\text{.}$

Worked example a training programme (illustration)

Participants earn $£8.80$ a week less: $\Delta=-£8.80\text{.}$ If participant selection contributes $-£12\text{,}$ then

\mathrm{ATT}=-8.80-(-12)=+£3.20\text{ a week.}

Why economists care

Closing a programme or justifying higher fees needs ATT, not $\Delta\text{.}$ Economists expect selection wherever people choose treatment on their prospects.

Check

$\Delta=12\text{.}$ Decompose it with $\mathrm{ATT}=5\text{,}$ and with $\mathrm{ATT}=15\text{.}$

Answer

$B=7$ and $B=-3\text{.}$ Both satisfy the identity; the data alone cannot choose.

Part 2 of 3

Designs that remove selection

How can the way treatment is assigned make selection bias zero, and for whom is the answer then valid?

Key terms

random assignmentITTcompliersLATEexternal validity

Notes 5.3, 5.4, 5.7 ↗

So far $\Delta=\mathrm{ATT}+B$ always holds, and the data alone cannot say how large $B$ is.

Why it matters

Randomised trials, offers and eligibility rules are the economist’s main evaluation tools. But people decline offers, effects spill over at scale and studies are short; each changes what is identified, and whether it carries over to a national policy.

In this part

5.3Random assignment: B = 0 on average
5.3, 5.7Compare by the offer: ITT, then LATE for compliers
▶Animation: the Wald ratio recovers the effect for compliers
5.4A credible trial need not transport
5.4Natural experiments and a threshold rule

5.3Random assignment

Random assignment sets selection bias to zero on average, not in every draw

Key formula Assumption (4)

Random assignment

D\perp\!\!\!\perp\{Y(0),Y(1)\}

In words assignment is independent of the pair of potential outcomes; then $B=0$ and $\Delta=\mathrm{ATT}=\mathrm{ATE}\text{.}$

Needs the design assigns treatment with the same positive probability to every unit.

Common mistake

It does not make the observed $Y$ independent of $D\text{:}$ a treatment effect is exactly how assignment changes $Y\text{.}$

All 35 ways to treat three of the seven people:

Unbiased over allocations; one draw can still be far off. Inference (Lecture 3) measures that chance variation.

5.3, 5.7Offers and take-up

People refuse offers: compare by the offer, then rescale for compliers

Key formula Definition (5)

Intention to treat

\mathrm{ITT}=E[Y\mid Z=1]-E[Y\mid Z=0]

In words $Z$ is the randomised offer, $D$ the treatment received. ITT is the effect of offering, take-up included.

Types by receipt without and with the offer, $(D(0),D(1))$
never-taker $(0,0)$	complier $(0,1)$
always-taker $(1,1)$	defier $(1,0)\text{:}$ ruled out

Common mistake

Comparing takers with non-takers: take-up selects on motivation, so it forfeits the randomisation.

Key formula Result (6)

The Wald ratio: LATE

\mathrm{LATE}=\frac{E[Y\mid Z=1]-E[Y\mid Z=0]}{E[D\mid Z=1]-E[D\mid Z=0]}

In words the offer’s effect on outcomes over its effect on take-up: the average effect for compliers.

Needs a random offer that affects $Y$ only through $D$ (exclusion), changes take-up, and creates no defiers.

LATE is local: a different offer moves different people, and its compliers may gain more or less than everyone else.

Final frame: two bars of people, without the offer (Z = 0) and with it (Z = 1), each split into always-takers 20%, compliers 50% and never-takers 30%. Without the offer only always-takers receive treatment (receipt 0.20); with it, always-takers and compliers do (receipt 0.70); dashed arrows mark the compliers who switch. Below, the gain in points: a dashed rectangle 3 high across everyone (+3 on average) and a solid rectangle of the same area, 6 high, under the compliers alone (+6 per complier). Beside it: random offer, the same mix in both groups; receipt 0.20 → 0.70; Δ_D = 0.50, the complier share; Δ_Y = 3 points over everyone; only compliers switch, so 0.50 × LATE = 3; LATE = Δ_Y/Δ_D = 3/0.50 = 6, the effect for compliers, not for everyone.

5.4Internal and external validity

A credible trial need not transport to another setting

Definition

Internal validity

Does the study credibly identify its stated effect for the units and treatment actually studied? Check assignment, take-up, measurement, missing outcomes, interference.

Definition

External validity

Does the result transport to another population, setting, implementation or time horizon? Check people, institutions, scale, follow-up.

Why a good trial may not carry over

Spillovers. If treating one person affects others, outcomes are $Y_i(d_i,\mathbf d_{-i})\text{:}$ a pilot and a national policy are different interventions.
Surrogates. An exam score next term is not earnings in ten years.
Being observed. Hawthorne and placebo effects: define the comparison; blind where possible.

Good random assignment supports internal validity without establishing external validity.

5.4Natural experiments

A natural experiment is an argument, not a label

Definition

Natural and quasi-experiments

Variation from outside the researcher’s control that resembles random allocation in the relevant respect. Name the source of variation, the comparison, the competing pathways and the local target.

Common mistake

“The weather was random.” A storm’s exposure varies with location and housing, and it can hit income directly, not only through moving house.

Worked example a class-size threshold rule

With classes capped at 40, enrolment $n$ predicts class size

q(n)=\frac{n}{\lceil n/40\rceil}

$n=40$	$40$	$n=80$	$40$
$n=41$	$20.5$	$n=81$	$27$

One more pupil drops the predicted size by 13. Compare schools just either side, if nothing else jumps at 40 and schools cannot sort.

Part 3 of 3

Adjusting for covariates

Without randomisation, when does comparing like with like recover the effect, and when does “controlling for” make things worse?

Key terms

CIAoverlaptarget weightsbad control

Notes 5.5–5.6 ↗

So far randomisation sets $B=0$ by design; a random offer identifies ITT and, under conditions, LATE for compliers.

Why it matters

Most economic data are observational, and adding controls is routine. Whether the gender pay gap “disappears within occupations” turns on exactly this question.

In this part

5.5CIA: as good as random within covariate cells
5.5Same cell effects, different targets
5.5No comparison without overlap
5.6Composition and the pay gap
5.6Control for confounders, not for outcomes of treatment
▶Animation: equal pay within a profession can hide a negative effect

5.5Selection on observables

Conditional independence: within covariate cells, assignment is as good as random

Key formula Assumption (7)

Conditional independence (CIA)

D\perp\!\!\!\perp\{Y(0),Y(1)\}\mid X

In words within each group of pre-treatment covariates $X\text{,}$ assignment is independent of the potential outcomes.

Needs an argument about how treatment was assigned. Then $\tau(x)=m_1(x)-m_0(x)\text{,}$ the within-cell difference of observed means.

What CIA does not say

CIA does not say potential outcomes are constant within a cell.
No number of controls makes it true, and balance on observed variables does not test it.

Worked example military service and earnings

Compare applicants who enlisted with applicants who did not, holding recorded screening characteristics fixed. Threat: persistence or reliable attendance may drive both enlistment and later earnings.

5.5Averaging cell effects

The same cell effects give different answers for different targets

Key formula Result (8)

Target weights

\begin{aligned}\mathrm{ATE}&=\textstyle\sum_x\tau(x)\,P(X=x)\\ \mathrm{ATT}&=\textstyle\sum_x\tau(x)\,P(X=x\mid D=1)\end{aligned}

In words ATE weights cells by the population; ATT by the treated.

Check

Cell effects 3 and 9; population shares 0.7 and 0.3; treated shares 0.2 and 0.8. ATE and ATT?

Answer

$\mathrm{ATE}=4.8\text{,}$ $\mathrm{ATT}=7.8\text{.}$

Worked example two cells, half the population each

Treatment probability $0.2$ in cell $X=0\text{,}$ $0.8$ in cell $X=1\text{.}$

Cell	Population	Treated	$\tau(x)$
$X=0$	0.5	0.2	2
$X=1$	0.5	0.8	6

\begin{aligned}\mathrm{ATE}&=0.5(2)+0.5(6)=4\\ \mathrm{ATT}&=0.2(2)+0.8(6)=5.2\end{aligned}

The raw gap, $23.2-12=11.2\text{,}$ adds baseline selection of $6\text{.}$

5.5Overlap and sparse cells

No comparison without overlap, and fine cells empty fast

Key formula Assumption (9)

Overlap (common support)

0<P(D=1\mid X=x)<1

In words at every covariate value in the target, some units are treated and some are not.

Needs for ATT, untreated comparisons wherever treated people are. It can be checked in the data.

Common mistake

If everyone with $X=1$ is treated, $\tau(1)$ is not identified: a fitted model fills the cell only by extrapolation, and trimming changes the estimand.

Worked example 100 points spread evenly over $k$ variables

Expected count in a cell of side $0.2$ of each range:

$k$	1	2	3	4	5
count	20	4	0.8	0.16	0.032

To expect 5 points with $k=5\text{,}$ each side must be $(0.05)^{1/5}\approx0.55$ of its range.

5.6Composition

Composition alone can make the overall gap exceed every within-group gap

Published statistic ONS 2021, provisional, UK medians

Median hourly-pay gaps: $7.9\%$ full-time, $-2.7\%$ part-time, $15.4\%$ overall. They compare pay across jobs, not people in the same job.

Common mistake

Medians cannot be averaged across groups: the law of iterated expectations is about means.

Worked example a hypothetical mean model

	Men	Women	Men’s share	Women’s share
Full-time	14.81	13.64	0.85	0.58
Part-time	9.07	9.31	0.15	0.42

Means $13.949$ and $11.8214\text{:}$ a gap of $15.25\%\text{,}$ above both within-stratum gaps ( $7.90\%$ and $-2.65\%$ ), because more women work in the lower-paid stratum.

5.6Bad controls

Control for confounders, not for outcomes of the treatment

A confounder: $X$ causes both $D$ and $Y\text{.}$ Conditioning on it can help.

A pre-treatment common cause

A bad control: $W$ is caused by $D\text{.}$ Holding it fixed changes who is compared.

Selection through an outcome

Key formula Identity (10)

Comparing within an occupation

\Delta_W=\tau_W+B_W

In words the gap among professionals = an effect for those who would be professional if treated + a difference in who becomes professional.

Needs $D$ independent of all potential earnings and occupations jointly.

Definition

Bad control

A variable caused by the treatment, such as occupation for pay. Conditioning on it can hide part of the effect or create selection.

Final frame: two columns of earnings, D = 0 and D = 1, for three types with shares 50%, 30% and 20%: type A earns 100 and 85, type B 60 and 45, type C 40 and 25, each pair joined by an arrow marked −15. Professional cells are ringed in gold: types A and B under D = 0, only type A under D = 1; the others are dimmed. Under each column: professionals, mean 85. Type A is outlined in both columns. Beside it: every type, Y(1) − Y(0) = −15; professional under D = 0: types A and B; under D = 1: type A only; D = 0: (0.5·100 + 0.3·60)/0.8 = 85; D = 1: 85, gap 85 − 85 = 0; τ_W = 85 − 100 = −15 for type A in both states; B_W = 100 − 85 = +15, because D = 0 adds type B; 0 = −15 + 15. Conditioning on a job changes who is compared.

Review · How the ideas connect

One identity, three ways to deal with selection

Part 1 · Effects

Part 2 · Designs that set B = 0

Part 3 · Adjustment

compare groups

Potential outcomes (1), targets ATE, ATT (2)

\Delta=\mathrm{ATT}+B

(3)
the selection-bias identity

B=0

on averagepartial take-up÷ take-up changefor whom?

Random assignment (4)

Random offer: ITT (5)

Compliers: LATE (6)

Does it transport?

B=0

in cellsreweight

\tau(x)

needed in every cellre-creates selection

CIA given

X

(7)

Weights for the target (8)

Overlap (9)

Bad control:

\Delta_W=\tau_W+B_W

(10)

Review · Summary sheet

What to take away: the terms and the ten formulas

Key terms

1 Effects and selection

Potential outcomes $Y_i(1)\text{,}$ $Y_i(0)\text{:}$ one unit’s outcomes with and without treatment; one is always missing.

ATE the average effect over everyone.

ATT the average effect among the treated.

Selection bias the groups’ difference in no-treatment outcomes.

2 Designs

Random assignment assignment independent of the potential outcomes.

complier takes the treatment if and only if offered.

LATE the average effect for the compliers of one instrument.

Internal and external validity credible for the units studied; carries over to other settings.

3 Adjustment

CIA within covariate cells, assignment independent of potential outcomes.

Overlap treated and untreated units at every target covariate value.

Bad control a variable the treatment changes; conditioning on it alters who is compared.

Key formulas

1 Effects and selection

(1)The observed outcome Identity

Y_i=D_iY_i(1)+(1-D_i)Y_i(0)

binary treatment, consistency, no interference

(2)Average treatment effects Definition

\mathrm{ATE}=E[\tau]\text{,}

\mathrm{ATT}=E[\tau\mid D=1]

each average over one stated population

(3)Observed difference = effect + selection Identity

\Delta=\mathrm{ATT}+B

always true when both groups exist

2 Designs

(4)Random assignment Assumption

D\perp\!\!\!\perp\{Y(0),Y(1)\}

gives

B=0\text{,}

\Delta=\mathrm{ATT}=\mathrm{ATE}

(population model)

(5)Intention to treat Definition

\mathrm{ITT}=E[Y\mid Z=1]-E[Y\mid Z=0]

identified when the offer

Z

is randomised

(6)The Wald ratio: LATE Result

\mathrm{LATE}=\Delta_Y/\Delta_D=E[Y(1)-Y(0)\mid C]

random offer, exclusion, nonzero take-up change, no defiers

3 Adjustment

(7)Conditional independence (CIA) Assumption

D\perp\!\!\!\perp\{Y(0),Y(1)\}\mid X

then

\tau(x)=m_1(x)-m_0(x)

where both groups exist

(8)Target weights Result

\begin{aligned}\mathrm{ATE}&=\textstyle\sum_x\tau(x)\,P(X=x)\\ \mathrm{ATT}&=\textstyle\sum_x\tau(x)\,P(X=x\mid D=1)\end{aligned}

CIA and overlap in every cell; discrete

X

(9)Overlap (common support) Assumption

0<P(D=1\mid X=x)<1

checkable; for every target covariate value

(10)Comparing within an occupation Identity

\Delta_W=\tau_W+B_W

given joint independence of D

Review · Before you report a causal estimate

Name the target, defend the assumption, then measure the uncertainty

Before you report a causal estimate

The intervention, population, outcome and time horizon.
How treatment was assigned.
The identifying condition: randomisation, exclusion, or CIA with overlap.
Only then, inference suited to the design. A confidence interval does not cover a failed assumption.

Result the opening question

From the data alone: cannot tell. In the model, every treated person gained (ATT $\approx+6.67$ ); their lower baselines ( $B\approx-23.33$ ) produced the gap.

The course in one line

Probability defines uncertainty; distributions and moments describe measurements; statistics studies sampling variation; time series accounts for dependence over time; causal inference decides which comparison answers an intervention question.

Practise

Notes 5.1–5.7 with worked solutions; the five interactive figures (potential outcomes, selection, the 35 allocations, target weights, pay mixtures).