Probability & Statistics · Lecture 3 of 5

Statistics

What can one sample tell us about a population, and how sure can we be?

1 Sampling2 Intervals3 Tests

Developed from Ian Crawford’s lecture material. Notes, slides and PDF: fatih.ai/probabilityandstatistics

Opening question

Smoking falls from 30% to 25% in both tables. Is the evidence the same?

100 people in a cessation programme, asked before and after; two constructed teaching tables with the same margins.

Table A: rows before, columns after
	smokes	not
smokes	20	10
not	5	65

Table B: rows before, columns after
	smokes	not
smokes	25	5
not	0	70

AA is stronger

BB is stronger

CThe same

Your answer

Vote now. Part 2 answers it.

Why start here

Before-and-after data on the same people are everywhere in programme evaluation. Whether a change is real depends on how the data were collected, not only on the headline percentages.

Lecture 3 · The route

From one sample to a range for the truth and a verdict on a claim

Part 1

Samples and sampling distributions

What can one sample tell us about a population, and how much would another sample differ?

populationiidestimatorstandard errorCLT

Notes 3.1–3.3

Part 2

Confidence intervals

How precisely can one sample pin down a mean, a proportion or a difference, and what does “95% confidence” promise?

confidence intervalcoveragecritical valuepaired data

Notes 3.4–3.5

Part 3

Hypothesis tests

When is a sample result strong enough evidence against a claim, and what do α, the p-value and “fail to reject” actually say?

null hypothesisType I errorpowerp-valueduality

Notes 3.6–3.8

By the end of this lecture you can

Distinguish a parameter, an estimator and an estimate, and say what the selection mechanism lets a sample tell us.
Derive the standard error $\sigma/\sqrt n$ , and use the LLN and CLT to justify an approximately normal estimate.
Construct and interpret 95% intervals for a mean, a proportion or a difference, matching the SE to the design.
Carry out a two-sided $z$ test and interpret $\alpha$ , power and the p-value correctly.

Part 1 of 3

Samples and sampling distributions

What can one sample tell us about a population, and how much would another sample differ?

Key terms

populationiidestimatorstandard errorCLT

Notes 3.1–3.3 ↗

Why it matters

Economists mostly work with samples: income surveys, polls and experiments. A sample average is informative only if the way units were selected targets the right population, and only as precise as its standard error.

In this part

3.1Who is sampled, and how
3.2Parameter, estimator and estimate
3.2Bias and the standard error
▶Animation: where the square root of n comes from
3.3The law of large numbers and the central limit theorem

3.1Target, design and iid

Say who is sampled and how before using any formula

Definition

Population and sample

The population is what we want to describe; the sample is what a stated selection mechanism delivers from a frame, the list units are drawn from.

Who is missing from the frame?
Who is selected, and who responds?
Which steps are random?

Common mistake

More respondents do not remove nonresponse bias: a very large sample from the wrong group answers the wrong question very precisely.

Definition

iid

Independent and identically distributed: the draws are jointly independent and each has the same distribution.

Worked example ten examination scripts, one graded 90

Draw two without replacement. Before any draw, $P(X_2=90)=1/10\text{,}$ the same as $X_1\text{.}$ But

P(X_2=90\mid X_1=90)=0\ne\tfrac1{10}.

Identical, yet dependent. With replacement, the draws are iid.

3.2Parameter, estimator, estimate

A parameter is fixed, an estimator is a random rule, an estimate is one number

Object	Random?	Example
Parameter $\theta$	fixed, unknown	population mean $\mu$
Estimator $\hat\theta$	random	$\bar X$
Estimate	a number	$\bar x=2.5$

Definition

Sampling distribution

The distribution of an estimator across repeated samples, not the spread of the data within one sample.

Key formula Definition (1)

Sample mean and variance

\bar X=\frac1n\sum_{i=1}^nX_i,\qquad s^2=\frac1{n-1}\sum_{i=1}^n(X_i-\bar X)^2

In words divide by $n-1\text{:}$ the $n$ deviations obey one constraint.

Worked example six observations: 1, 2, 1, 3, 4, 4

$\bar x=15/6=2.5\text{;}$ the squared deviations sum to $9.5\text{,}$ so

s^2=9.5/5=1.9,\qquad s\approx1.378.

3.2Bias and standard error

Judge an estimator by its centre and its spread: bias and standard error

Definition

Bias

$\operatorname{Bias}(\hat\theta)=E[\hat\theta]-\theta\text{,}$ averaged over repeated samples. $\bar X$ is unbiased: $E[\bar X]=\mu$ needs only common means.

Key formula Result (2)

Standard error of the mean

\operatorname{Var}(\bar X)=\frac{\sigma^2}{n},\qquad\operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}

In words estimated by $s/\sqrt n\text{.}$

Needs independent draws with a common variance; dependence adds covariance terms.

Worked example one score against the mean of 20

Scores uniform on 0, …, 100: $\sigma^2=850\text{,}$ so one score has SD $\sqrt{850}\approx29.15\text{.}$ The mean of 20 independent scores has

\mathrm{SE}=\sqrt{850/20}=\frac{29.15}{\sqrt{20}}\approx6.52.

Common mistake

The SD describes individuals; the standard error describes the estimator across samples. For 1, 2, 1, 3, 4, 4: $s\approx1.378$ but $\widehat{\mathrm{SE}}\approx0.563$ .

Final frame: a 20-by-20 grid with the same small yellow amount in every cell, labelled “spread evenly: 42.5 per cell”. Below, on a 0 to 100 axis, a flat band for one score with a bracket of ±29.15, and the much narrower exact distribution of the mean of 20 scores with a bracket of ±6.52. Beside them: Var(X₁ + ⋯ + X₂₀) is the sum of all 400 cells, = 20 × 850 = 17000; Var(X̄) = 17000/20² = 42.5, the average cell, 850/20; X_i is uniform on 0, …, 100 with σ² = 850; SE(X̄) = √42.5 = 29.15/√20 ≈ 6.52.

3.3Large samples

Averages concentrate, and their errors become approximately normal

Result the law of large numbers

For iid data with finite mean and variance, $\bar X_n$ converges in probability to $\mu\text{:}$ $P(|\bar X_n-\mu|>\varepsilon)\to0\text{.}$

Key formula Result (3)

Central limit theorem

\frac{\bar X_n-\mu}{\sigma/\sqrt n}\xrightarrow{d}N(0,1)

In words for large $n\text{,}$ $\bar X_n$ is approximately $N(\mu,\sigma^2/n)\text{,}$ and $s$ may replace $\sigma\text{.}$

Needs iid data with $0<\sigma^2<\infty\text{.}$

Worked example a U-shaped population

A Beta $(\tfrac12,\tfrac12)$ variable has SD $0.3536\text{.}$ Means of 50 have SE $\sqrt{(1/8)/50}=0.05$ and a bell-shaped histogram; at $n=200$ the SE halves to $0.025\text{.}$ The data stay U-shaped.

Common mistake

The CLT is about means, not the data.
There is no universal “ $n=30$ ”.
No sample size cures a biased selection.

An estimate is one draw from a sampling distribution: centred on its target if unbiased, with spread $\sigma/\sqrt n\text{.}$

Part 2 of 3

Confidence intervals

How precisely can one sample pin down a mean, a proportion or a difference, and what does “95% confidence” promise?

Key terms

confidence intervalcoveragecritical valuepaired data

Notes 3.4–3.5 ↗

So far an estimate is one draw from a sampling distribution; its standard error $\sigma/\sqrt n$ measures the spread, and the CLT makes it approximately normal.

Why it matters

Decisions need a range, not a point. A 39% approval rate means little without its 2.47-point margin, and a fall in smoking after a programme is convincing or not depending on how many people actually changed.

In this part

3.4A confidence interval and its coverage
▶Animation: what the 95% in a confidence interval describes
3.4One recipe, three designs
3.5A proportion is a mean of 0/1 data
3.5The opening question answered

3.4Building an interval

A confidence interval turns a statement about the estimate into a range for the target

Key formula Result (4)

Confidence interval for a mean

\bar X\pm c\,\mathrm{SE}

In words $P(\bar X-c\,\mathrm{SE}\le\mu\le\bar X+c\,\mathrm{SE})=1-\alpha\text{:}$ the random interval covers the fixed $\mu$ with probability $1-\alpha\text{.}$

Needs exact for normal data with known $\sigma\text{;}$ approximate for large samples with $s/\sqrt n\text{.}$

Definition

Coverage

The share of repeated samples whose interval contains the parameter. It belongs to the procedure, not to one computed interval.

Critical values
Confidence	90%	95%	99%
$c$	1.645	1.960	2.576

Result width

The width is $2c\,\mathrm{SE}\text{:}$ more confidence widens it; four times the sample size halves it.

The algebra moves the randomness to the endpoints; $\mu$ stays fixed.

Final frame: the sampling distribution of the mean, N(0, 1/5), with its central 95% between −0.877 and 0.877 shaded and its two 2.5% tails in red; dashed lines continue the band’s edges down the page, and a fixed dashed line marks μ = 0. Below, 40 simulated intervals x̄ ± 0.877: those whose mean lies inside the band cross the μ line (green); 3 whose mean lies outside miss it (red). Beside it: n = 5 from N(0, 1), X̄ ~ N(0, 1/5); 1.96/√5 ≈ 0.877, so P(|X̄ − μ| ≤ 0.877) = 0.95; each sample gives the interval x̄ ± 0.877; 1.00 ± 0.877 = [0.123, 1.877] misses 0; a miss happens exactly when x̄ lands outside the 95% band; this run: 37 of 40 contain 0; 95% describes the procedure over many samples, and one computed interval either contains μ or not.

3.4Means in three designs

One recipe, three designs: the design decides the standard error

Design	Example	Estimate	SE	95% interval
One mean	$n=100\text{,}$ $\bar x=64\text{,}$ $s^2=8$	$64$	$0.283$	$[63.446,\ 64.554]$
Independent groups	anchoring prompt 10 vs 65 (43 and 46 students)	$-13.6$	$\sqrt{7.5}\approx2.739$	$[-18.968,\ -8.232]$
Paired	71 students, exam 1 minus exam 2	$-1.8$	$\sqrt{124.45/71}\approx1.324$	$[-4.395,\ 0.795]$

Key formula Result (5)

Standard errors by design

\sqrt{\frac{s_X^2}{n_X}+\frac{s_Y^2}{n_Y}}\quad\text{or}\quad\frac{s_D}{\sqrt n}

In words independent groups: variances add. Paired: analyse the $n$ differences, $s_D^2=s_X^2+s_Y^2-2s_{XY}\text{.}$

Common mistake

Treating paired scores as independent: $\sqrt{117/71+151/71}$ ignores the positive covariance and overstates the SE. And $[-4.395,0.795]$ including 0 does not prove the means equal.

3.5Proportions

A proportion is a mean of 0/1 data, with its own standard error

Key formula Result (6)

Interval for a proportion (Wald)

\hat p\pm c\sqrt{\frac{\hat p(1-\hat p)}{n}}

In words $\operatorname{Var}(\hat p)=p(1-p)/n\text{,}$ estimated by plugging in $\hat p\text{.}$

Needs a large iid sample, away from 0 and 1.

Common mistake

At 0 successes in 10 the Wald interval is $[0,0]\text{.}$ Clipping does not repair it; the Wilson interval, about $[0,0.2775]\text{,}$ does better.

Worked example an approval survey

$n=1502\text{,}$ $\hat p=0.39\text{,}$ treated as an iid unweighted sample:

\mathrm{SE}=\sqrt{\frac{0.39(0.61)}{1502}}\approx0.0126

$0.39\pm1.96(0.0126)\text{:}$ from $36.53\%$ to $41.47\%\text{,}$ a margin of about $2.47$ percentage points, not $2.47\%\text{.}$

3.5Paired proportions

Same fall from 30% to 25%, different uncertainty: the people who changed decide

Each person’s change $D_i$ is $-1$ (quit), $0$ (no change) or $+1$ (start). The SE is $s_D/\sqrt{n}\text{,}$ and only people who changed contribute to $s_D\text{.}$

Table A: 10 quit, 5 start: rows before, columns after
	smokes	not
smokes	20	10
not	5	65

Table B: 5 quit, 0 start: rows before, columns after
	smokes	not
smokes	25	5
not	0	70

Worked example the two tables

Table A: $s_D^2=14.75/99\text{,}$ SE $\approx0.0386\text{:}$

-0.05\pm1.96(0.0386)\approx[-0.126,\ 0.026]

Table B: $s_D^2=4.75/99\text{,}$ SE $\approx0.0219\text{:}$

-0.05\pm1.96(0.0219)\approx[-0.093,\ -0.007]

B’s interval rests on only five changes, so an exact test is safer; and a before–after fall is not, by itself, the programme’s effect.

The interval is estimate ± critical value × SE, with the SE matched to the design: one sample, independent groups or pairs.

Part 3 of 3

Hypothesis tests

When is a sample result strong enough evidence against a claim, and what do α, the p-value and “fail to reject” actually say?

Key terms

null hypothesisType I errorpowerp-valueduality

Notes 3.6–3.8 ↗

So far a 95% interval is the estimate $\pm1.96$ standard errors, with the standard error matched to the design.

Why it matters

Economists test claims and act on them: an evenly split electorate, no anchoring effect, no change after a programme. Reading a p-value as the probability that a claim is true, or “not significant” as “no effect”, produces false findings.

In this part

3.6What a test controls
3.6One test from start to finish
▶Animation: what a p-value is an area of
▶Animation: what a stricter level costs in power
3.7Tests for means and proportions
3.8A confidence interval is the set of nulls not rejected

3.6Hypotheses and error rates

A test limits how often it rejects a true claim, not the chance that the claim is true

State $H_0:\theta=\theta_0$ against $H_1:\theta\ne\theta_0$ about a parameter, with the direction and level fixed in advance.

	$H_0$ true	$H_0$ false
Reject	Type I error	correct: power
Do not reject	correct	Type II error

Key formula Definition (7)

Level and power

P(\text{reject }H_0\mid H_0\text{ true})\le\alpha

In words power $1-\beta(\theta)$ is the probability of rejecting at a given alternative.

Check

A 5% test rejects. Is there now a 5% chance that $H_0$ is true?

Answer

No: $\alpha$ bounds $P(\text{reject}\mid H_0)\text{,}$ not $P(H_0\mid\text{reject})\text{.}$ Reversing the condition needs Bayes’ rule (Lecture 1).

Many tests

Run 20 tests of true nulls at 5% and $1-0.95^{20}\approx0.64$ is the chance of at least one false finding. Bonferroni uses $\alpha/m\text{.}$

3.6The statistic and the p-value

Count the standard errors from the claim; the p-value is the tail beyond them

Key formula Definition (8)

Test statistic

Z=\frac{\hat\theta-\theta_0}{\widehat{\operatorname{SE}}_0(\hat\theta)}

In words the distance from the claim in null standard errors; reject at 5% if $|Z|>1.96\text{.}$

Key formula Definition (9)

Two-sided p-value

p_{\rm val}=2\{1-\Phi(|z_{\rm obs}|)\}

In words the probability, if $H_0$ were true, of a statistic at least as extreme as the one observed.

Worked example $H_0:\mu=64.5$

$\bar x=64\text{,}$ SE $\sqrt{8/100}\approx0.283\text{.}$
$z=(64-64.5)/0.283\approx-1.77\text{.}$
$p_{\rm val}=2\{1-\Phi(1.77)\}\approx0.077\text{.}$
$0.077>0.05\text{:}$ fail to reject. Report $64\text{,}$ $[63.45,64.55]$ and $p_{\rm val}\approx0.077\text{.}$

Common mistake

“We failed to reject, so $\mu=64.5\text{.}$ ” The data are just as compatible with 64.3.

Final frame: the null sampling distribution N(64.5, 0.2828²) of the mean grade, with a z scale below it. The observed mean 64 (z ≈ −1.77) and its mirror image 65 are marked in yellow; the two tails beyond them are shaded red, 0.0385 each. Red lines mark the 5% cutoffs z = ±1.96, further out than the observed value. Beside it: H₀: μ = 64.5, n = 100, s² = 8; SE = √(8/100) ≈ 0.2828, x̄ = 64; z = (64 − 64.5)/0.2828 ≈ −1.77; p = P(|Z| ≥ 1.77 | H₀) = 2{1 − Φ(1.77)} ≈ 0.077; 0.077 > 0.05, so fail to reject H₀ at 5%, equivalently |z| = 1.77 < 1.96. The p-value assumes H₀; it is not the probability that H₀ is true.

Final frame: on a z axis, the null curve N(0, 1) and the true curve N(2.5, 1). Yellow cutoff lines at ±2.576; the null’s red tails beyond them have total area α = 0.010; the true curve’s green area beyond 2.576 is the power, 0.470; a yellow strip between 1.96 and 2.576 marks the 0.235 of power given up by moving from 5% to 1%. Beside it: H₀ true: Z ~ N(0, 1); true mean 2.5 SE above: Z ~ N(2.5, 1); at 5%, reject if |Z| > 1.96, power = Φ(2.5 − 1.96) = Φ(0.54) ≈ 0.705; at 1%, reject if |Z| > 2.576, power ≈ Φ(2.5 − 2.576) = Φ(−0.076) ≈ 0.470. A smaller Type I error bound costs power, unless the sample size rises.

3.7Tests by design

The same designs as the intervals, with the standard error computed under the null

Design	$H_0$	SE under $H_0$	$z$	$p_{\rm val}$	At 5%
One mean	$\mu=65$	$s/\sqrt n=0.283$	$-3.54$	$0.0004$	reject
Independent means	no anchoring effect	$\sqrt{7.5}$	$-4.97$	$<0.0001$	reject
Paired means	equal exam means	$\sqrt{124.45/71}$	$-1.36$	$0.17$	do not reject
One proportion	$p=0.5$	$\sqrt{p_0(1-p_0)/n}$	$-8.53$	$<0.0001$	reject
Two proportions	$p_X=p_Y$	pooled $\hat\pi\approx0.390$	$3.58$	$0.0003$	reject
Paired proportions	no change (Table A)	$s_D/\sqrt n$	$-1.30$	$0.20$	do not reject

Common mistake

For proportions the null fixes the SE: use $p_0\text{,}$ not $\hat p\text{,}$ for one proportion, and pool only when testing equality.

3.8Tests and intervals agree

An interval is every claim the matching test keeps

Key formula Result (10)

Inverting a test

\Bigl|\frac{\hat\theta-\theta_0}{e}\Bigr|\le c\iff\theta_0\in[\hat\theta-c e,\ \hat\theta+c e]

In words a value is retained by the test exactly when it lies in the interval.

Needs the same data, SE $e\text{,}$ reference distribution, tails and level.

Report the estimate and its interval, not just “significant”: significant is not important, and not rejecting is not equivalence.

Worked example $\bar x=64\text{,}$ SE $0.283\text{,}$ interval $[63.446,64.554]$

$\mu_0=64.2\text{:}$ $z\approx-0.71\text{,}$ $p_{\rm val}\approx0.48\text{,}$ retained.
$\mu_0=65\text{:}$ $z\approx-3.54\text{,}$ $p_{\rm val}\approx0.0004\text{,}$ rejected.

Common mistake

For a proportion the score test (SE from $p_0$ ) inverts to the Wilson interval, not the Wald interval: only a matching pair agrees.

A test counts null standard errors from the claim; the interval is every claim it would not reject.

Review · How the ideas connect

One chain: design, estimator, standard error, then interval and test

Part 1 · Sampling

Part 2 · Intervals

Part 3 · Tests

apply a rulerepeat the samplestandardise

Design: frame, selection, response

Estimator

\hat\theta

(1)

Sampling distribution: SE (2)

CLT (3):

\hat\theta\approx

normal

the design sets the SEplug ininvert0/1 data

SE by design: one, independent, paired (5)

Interval

\hat\theta\pm c\,\mathrm{SE}

(4)

Proportions (6)

evaluate at

\theta_0

truth vs decisionretained valuesthe same set

Test:

Z\text{,}

p_{\rm val}

(8) (9)

Error rates:

\alpha\text{,}

power (7)

Duality (10)

Review · Summary sheet

What to take away: the terms and the ten formulas

Key terms

1 Sampling

population the collection or distribution we want to make a statement about.

iid independent draws, each with the same distribution.

Estimator $\hat\theta=g(X_1,\dots,X_n)\text{:}$ a rule applied to the random sample.

Sampling distribution how an estimator varies across repeated samples.

Bias $E[\hat\theta]-\theta\text{:}$ the average error over repeated samples.

standard error the SD of an estimator’s sampling distribution.

2 Intervals

Coverage the share of repeated samples whose interval contains the parameter.

Critical value $c=z_{1-\alpha/2}\text{:}$ 1.645, 1.960, 2.576 for 90, 95, 99%.

3 Tests

Type I error rejecting a true null hypothesis.

Key formulas

1 Sampling

(1)Sample mean and variance Definition

\bar X=\frac1n\sum_i X_i\text{;}

s^2

divides by

n-1

iid, finite moments,

n>1

(2)Standard error of the mean Result

\operatorname{Var}(\bar X)=\frac{\sigma^2}{n},\qquad\operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}

independence and identical variances

(3)Central limit theorem Result

\frac{\bar X_n-\mu}{\sigma/\sqrt n}\xrightarrow{d}N(0,1)

iid,

0<\sigma^2<\infty

2 Intervals

(4)Confidence interval for a mean Result

\bar X\pm c\,\mathrm{SE}

coverage

1-\alpha\text{;}

large samples use

s/\sqrt n

(5)Standard errors by design Result

\sqrt{\frac{s_X^2}{n_X}+\frac{s_Y^2}{n_Y}}\quad\text{or}\quad\frac{s_D}{\sqrt n}

independence between groups, or between pairs

(6)Interval for a proportion (Wald) Result

\hat p\pm c\sqrt{\frac{\hat p(1-\hat p)}{n}}

large iid sample, away from 0 and 1

3 Tests

(7)Level and power Definition

P(\text{reject }H_0\mid H_0\text{ true})\le\alpha

fix the level and direction before the data

(8)Test statistic Definition

Z=\frac{\hat\theta-\theta_0}{\widehat{\operatorname{SE}}_0(\hat\theta)}

approximately

N(0,1)

under

H_0

(9)Two-sided p-value Definition

p_{\rm val}=2\{1-\Phi(|z_{\rm obs}|)\}

computed under

H_0\text{;}

one tail only if chosen in advance

(10)Inverting a test Result

\Bigl|\frac{\hat\theta-\theta_0}{e}\Bigr|\le c\iff\theta_0\in[\hat\theta-c e,\ \hat\theta+c e]

same SE, reference distribution, tails and level

Review · Back to the opening question

Same 5-point fall; only Table B’s interval excludes zero

Result the opening question

The margins cannot decide it: the standard error comes from the people who changed. Table A, 15 changes: $[-0.126,0.026]\text{.}$ Table B, 5 changes: $[-0.093,-0.007]\text{.}$

Remember

An estimate is one draw; its SE says how far another would land.
95% describes the procedure, not one computed interval.
A p-value is computed assuming $H_0\text{;}$ it is not the probability that $H_0$ is true.

Before next time

Notes 3.1–3.8 with the worked solutions.
The sampling, coverage, tails and duality figures in the notes.
Rework the six tests in the table for yourself.

Next · Lecture 4

Time series. When observations depend on their past, they are not iid: what changes for models, forecasts and standard errors?