3.1.1 What are we trying to learn about?

The sampling frame is the operational list from which units can be selected. An electoral register may be a frame for registered electors; it is not automatically a complete frame for every resident. Coverage gaps between the frame and the target population must be considered separately from random sampling variation.

3.1.2 Random selection and representativeness

For a fixed-size simple random sample without replacement, every subset of $n$ distinct population units has the same chance of selection. Consequently, each unit has inclusion probability $n/M$. Equal individual inclusion probabilities alone are a weaker property: a design could always select whole clusters and still give each person the same chance.

Random selection makes uncertainty assessable under the design. It does not guarantee that one realised sample exactly matches every population feature. A sample mean may be above or below the population mean. The frequency and size of such discrepancies depend on the design, sample size and population variation; “random” by itself supplies no numerical assurance.

In quota sampling, recruitment is arranged to match specified characteristics. For instance, a hypothetical population with $20\%$ graduates would yield a graduate quota of $20$ in a sample of $100$; a subgroup making up $5\%$ would have quota $5$. Meeting these quotas does not make selection within each subgroup random. People who are easy to contact may differ in income or attitudes from those who are hard to contact.

Even a probability sample can lose its intended properties through nonresponse. If high-income people are systematically less likely to answer an income survey, the responding sample may understate the population mean. Increasing the number of respondents does not automatically remove that bias. Frame quality, recruitment, response and measurement all contribute to the inference.

Before calculating: Who is absent from the frame? Who is selected? Who responds? Which of these steps is random, and does the reported standard error account for the actual design?

3.1.3 A random sample and its realised values

Examination scripts: why replacement matters

Consider ten scripts with grades $30,30,30,60,60,80,80,80,90,100$. Select a script uniformly. The grade of the first draw has the following distribution. Now suppose the first selected grade is $90$, and it is not replaced.

The right column is conditional on the first result. Before observing either draw, $X_2$ has the same marginal distribution as $X_1$: every script is equally likely to occupy the second position. The draws therefore have identical marginal distributions but are dependent. Independence fails because, for example, $P(X_2=90\mid X_1=90)=0\ne1/10$.

If the script is replaced and each selection is made independently, every subsequent draw again has the original probabilities. This mechanism supplies both identical distributions and independence. With $1{,}000$ scripts in the same grade proportions, removing a grade-90 script changes the conditional probabilities to $300/999$, $200/999$, $300/999$, $99/999$, $100/999$. The change after one draw is small. Across a large sample, however, the sampling fraction $n/M$ matters; a large population alone does not justify ignoring dependence.

3.1.4 Check the mechanism

3.2.1 What changes across samples?

3.2.2 Sample analogues of population moments

The population mean is a probability-weighted average of possible values. Its sample analogue gives the observed values equal weights. For iid data with finite moments, use

The realised sample $1,2,1,3,4,4$ has sum $15$, size $6$, and mean $2.5$. Its sum of squared deviations is $9.5$, so its sample variance is $9.5/5=1.9$ and sample SD is $\sqrt{1.9}\approx1.378$. Squaring deviations prevents positive and negative deviations cancelling; division sets the variance's scale.

For binary indicators $X_i\in\{0,1\}$, the mean is the proportion of ones. Thus $0,1,0,1,1$ gives $\hat p=3/5=0.6$. A Bernoulli population has mean $p$ and variance $p(1-p)$. Two related sample quantities are

The first estimates the population variance by substituting the estimated probability. The second is the usual sample variance with the unbiased denominator. Here they are $0.24$ and $0.30$, respectively. They approach each other as $n$ grows, but they are not identical.

Covariance needs information about pairs

The population covariance is $\sigma_{XY}=E[(X-\mu_X)(Y-\mu_Y)]$; when both population SDs are positive, population correlation is $\rho_{XY}=\sigma_{XY}/(\sigma_X\sigma_Y)$. Read $\rho$ as “rho.” For paired measurements $(X_i,Y_i)$, sample covariance measures whether deviations tend to share a sign:

Read $s_{XY}$ as “sample covariance of X and Y” and $r_{XY}$ as “sample correlation of X and Y.” Correlation is defined only when both sample SDs are positive. Covariance carries the product of the two variables' units; correlation has no units.

For two binary variables, writing $p_X=P(X=1)$, $p_Y=P(Y=1)$, and $q=P(X=1,Y=1)$,

The product of marginal probabilities is not the covariance. Independence would make $q=p_Xp_Y$ and covariance zero. With observed joint proportion $\hat q$, the sample covariance is $n(\hat q-\hat p_X\hat p_Y)/(n-1)$. Marginal percentages alone cannot recover it. This fact will determine the correct uncertainty for before-and-after proportions.

3.2.3 Bias and the variance denominator

Definition. $\operatorname{Bias}(\hat\theta)=E[\hat\theta]-\theta$. An estimator is unbiased if this difference is zero.

Unbiasedness concerns an average over repeated samples. It does not mean every realised estimate equals the target. If each observation has finite expectation $\mu$, linearity gives

This proof needs common expectations, not independence. Independence will matter for the variance of the mean. A histogram of means from samples of $20$ drawn from a discrete uniform population $1,\ldots,100$ should centre near $50.5$. Ten illustrative simulated means are $46.10$, $55.15$, $58.10$, $52.70$, $41.30$, $52.00$, $53.35$, $49.10$, $53.50$, $46.75$. Their variation is compatible with an unbiased rule; a finite simulation need not centre exactly on the parameter.

Why divide squared deviations by $n-1$?

1. Separate the unknown true centre from the fitted centre. Expanding the squares and using $\sum_i(X_i-\bar X)=0$ gives $$\begin{aligned}\sum_i(X_i-\bar X)^2&=\sum_i(X_i-\mu)^2\\ &\quad-n(\bar X-\mu)^2.\end{aligned}$$
2. Take expectations under iid finite-variance sampling. The first sum has expectation $n\sigma^2$. Since $E\bar X=\mu$ and $\operatorname{Var}(\bar X)=\sigma^2/n$, the second term has expectation $\sigma^2$.
3. Correct the systematic shortfall. The expected sum is $(n-1)\sigma^2$. Dividing by $n$ therefore gives expectation $(n-1)\sigma^2/n$, below the target. Dividing by $n-1$, for $n>1$, gives $E[s^2]=\sigma^2$.

Estimating the centre from the same data makes deviations artificially small: the fitted mean minimises their squared sum. The $n$ residuals obey one linear constraint, leaving $n-1$ degrees of freedom. They are not themselves independent observations. Also, an unbiased variance does not make its square root an unbiased SD: averaging and taking square roots do not commute.

3.2.4 Standard deviation versus standard error

Without independence the general expression also includes $2\sum_{i<j}\operatorname{Cov}(X_i,X_j)/n^2$. Positive dependence can increase uncertainty relative to an iid calculation. The next topics explain why time dependence and sampling design cannot be ignored.

For the discrete uniform population $0,\ldots,100$, $\mu=50$ and $\sigma^2=850$. Mean variances are $850/20=42.5$ and $850/40=21.25$. An illustrative set of $100$ replications gave empirical variances $45.24$ and $23.64$. These are finite simulation results, not alternative theoretical identities. Doubling sample size halves variance; quadrupling it halves SE.

1. One score is uniform on $0,\dots,100\text{,}$ with variance $\sigma^2=850$ and SD $\approx29.15\text{.}$
2. The variance of a sum of $20$ scores adds up all $20\times20=400$ cells of a grid, where cell $(i,j)$ is $\operatorname{Cov}(X_i,X_j)\text{.}$
3. Independent scores have zero covariance, so only the $20$ diagonal cells remain, each $850\text{:}$ the sum has variance $20\times850=17000\text{.}$
4. The mean divides the sum by $20\text{,}$ so its variance is divided by $20^2=400\text{:}$ spread evenly, $42.5$ per cell, so $\operatorname{Var}(\bar X)=850/20\text{.}$
5. So $\operatorname{SE}(\bar X)=\sqrt{42.5}=29.15/\sqrt{20}\approx6.52\text{:}$ a mean of $20$ scores varies far less than one score.

3.2.5 Efficiency and consistency

Among unbiased estimators of the same target under the same model, the one with smaller sampling variance is more efficient. A noisy sample median and a sample mean may both converge to $50$ for the symmetric uniform population $0,\ldots,100$. That example does not establish a universal ranking, and a median generally targets a population median rather than a mean.

Definition. $\hat\theta_n$ is consistent for $\theta$ if, for every $\varepsilon>0$, $P(|\hat\theta_n-\theta|>\varepsilon)\to0$ as $n\to\infty$.

Read this as “the probability of an error larger than epsilon tends to zero.” It is a statement about a sequence of sampling distributions, not a promise that a realised estimate moves closer at every step.

A useful sufficient condition is vanishing mean squared error:

The inequality follows by applying Markov's inequality to squared error. Thus bias tending to zero and variance tending to zero imply consistency. This is sufficient, not a necessary characterisation of all consistent estimators.

Consistency does not require finite-sample unbiasedness. For iid draws uniformly from $0,\ldots,1000$, use their maximum to estimate the population maximum $1000$. It never exceeds the target, and has positive probability of falling short for every finite $n$, so it is biased downwards. Yet $P(\max_i X_i<1000)=(1000/1001)^n\to0$. With a nested growing sample, the maximum can only stay the same or rise; it cannot bounce downwards.

3.3.1 One sample is not a sampling distribution

A histogram of $n$ observations depicts the observed data. A histogram of $R$ sample means, each calculated from a separate sample of size $n$, depicts a simulation of the mean's sampling distribution. Keep those two counts separate: $n$ is observations per sample; $R$, read “capital R,” is the number of repeated samples.

The parameter stays fixed across this thought experiment. The estimator varies because each sample changes. After one sample has been realised, its estimate is fixed too; its standard error describes the rule's uncertainty across other possible samples under the model.

For independent uniform draws on $0,\ldots,100$, individuals have SD $\sqrt{850}\approx29.15$. The mean from $20$ draws has SE $\sqrt{850/20}\approx6.52$. Individuals can remain widely spread even when their averages are concentrated. This separation is the basis of statistical inference.

3.3.2 The law of large numbers

The bound provides a conservative finite-sample statement when the variance is known. With variance $850$, sample size $1{,}000$, and tolerance $5$, the error probability is at most $850/(1000\times25)=0.034$. The LLN itself is a limiting result; it does not specify that every chosen finite sample is already accurate.

A running mean may move away from its target temporarily as new observations arrive. That does not contradict convergence. Nor does more data remove selection bias: if the sample comes from the wrong distribution, the average can converge very precisely to the wrong target.

3.3.3 The central limit theorem

A strongly non-normal parent

A Beta distribution with parameters $1/2,1/2$ is U-shaped on $(0,1)$, with mean $1/2$ and variance $1/8$. Individual samples retain that U-shaped tendency. Means from $1{,}000$ independent samples of $50$ observations can have a bell-shaped sampling histogram even though the individual measurements do not.

Standardising each mean as $(\bar X_{50}-0.5)/0.05$ allows comparison with a standard normal curve. Neither a bell-shaped simulation nor agreement with a curve proves the theorem. The assumptions and mathematical limit do that work; the simulation illustrates the distinct distributions involved.

3.3.4 From the theorem to an estimated SE

The CLT denominator contains $\sigma$, often unknown. If $s$ consistently estimates $\sigma>0$, replacing the population SD by the sample SD preserves the asymptotic normal limit. This substitution is justified by the combination of consistency and Slutsky's theorem; it does not make the finite-sample distribution exactly normal.

For iid Bernoulli data, $\mu=p$ and $\sigma^2=p(1-p)$. The mean is a proportion, so its estimated SE is $\sqrt{\hat p(1-\hat p)/n}$. Normal approximation can be poor when expected successes $np$ or failures $n(1-p)$ are small. Values near the boundaries require particular care.

Variance is a nuisance parameter when our target is the mean: it is not the main object of interest, but it governs the uncertainty calculation. Estimating it matters especially with small samples. The next chapter distinguishes exact normal/known-variance inference, normal/unknown-variance inference using Student's $t$, and large-sample approximations.

3.4.1 What does confidence describe?

A point estimate gives one answer. A confidence interval adds a range constructed by a procedure whose coverage can be assessed across repeated samples. The population parameter remains fixed; the interval endpoints change when the sample changes.

If we repeatedly draw samples of size $5$ from $N(0,1)$, the known-SD intervals $\bar X\pm1.96/\sqrt5$ have approximately $95\%$ coverage. Among $20$ repetitions, the number covering is random, not guaranteed to be $19$. A single observed interval cannot reveal its own coverage status when the true mean is unknown.

1. Samples of $n=5$ from $N(0,1)\text{:}$ the mean $\bar X\sim N(0,1/5)$ lands within $1.96/\sqrt5\approx0.877$ of the fixed $\mu=0$ with probability $0.95\text{.}$
2. Each sample gives its own interval $\bar x\pm0.877\text{.}$ The means $-0.10$ and $0.40$ give intervals that contain $0\text{.}$
3. The mean $1.00$ lands outside the central $95\%\text{,}$ so its interval $[0.123,1.877]$ misses $0\text{.}$ An interval misses exactly when its mean lands outside that band.
4. Forty simulated samples: in this run $37$ of the $40$ intervals contain $0\text{.}$ The $95\%$ describes the procedure over many samples; one computed interval either contains $\mu$ or does not.

3.4.2 Deriving a mean interval

Start with a statement about a statistic whose distribution we know or approximate. For iid normal observations with known $\sigma>0$, $Z=(\bar X-\mu)/(\sigma/\sqrt n)$ is exactly standard normal. Let $c=z_{1-\alpha/2}$ be the quantile with probability $\alpha/2$ above it. Symmetry puts the same probability below $-c$.

1. Central probability. $P(-c\leq Z\leq c)=1-\alpha$.
2. Rescale by a positive number. Multiply the inequalities by $\sigma/\sqrt n$, obtaining $-c\sigma/\sqrt n\leq\bar X-\mu\leq c\sigma/\sqrt n$.
3. Solve for the target. Subtract $\bar X$, reverse the inequalities when multiplying by $-1$, and reorder:

The algebra has not made the unknown mean random. It has turned a probability statement about a random estimator into one about random endpoints containing a fixed mean. Read $\pm$ as “plus or minus”: it specifies both endpoints.

For $90\%,95\%,99\%$ confidence, the standard normal critical values are approximately $1.645,1.960,2.576$. Higher confidence requires a wider interval at the same SE. Width is $2c\operatorname{SE}$; quadrupling an iid sample size halves width when population variation is held fixed. Extra precision and extra confidence are different objectives.

Which critical value belongs to which model?

With an unknown SD and a suitable large sample, use $\bar X\pm c s/\sqrt n$ as an approximate interval. With iid normal data and unknown SD, the exact pivot is instead

Read $t_{n-1}$ as “t with n minus one degrees of freedom.” Student's $t$ distribution has heavier tails, accounting for uncertainty in the SD estimate. Use its quantile $t_{n-1,1-\alpha/2}$ in place of $c$. This exact small-sample result needs normality, not just independence. For non-normal small samples, neither normal nor $t$ calibration is automatically reliable.

3.4.3 One mean: a complete calculation

A sample has $n=100$, observed mean $\bar x=64$, and sample variance $s^2=8$. Under the iid large-sample model, estimate the uncertainty in the population mean.

1. Identify the estimator. $\bar X$ estimates $\mu$, with realised value $64$.
2. Convert data spread into uncertainty of the mean. $s=\sqrt8$; estimated SE is $\sqrt{8/100}\approx0.28284$. The number $8$ is a variance, not an SD.
3. Choose the confidence level. At $95\%$, use $c\approx1.96$. Margin $1.96\times0.28284\approx0.55437$.
4. Report endpoints and conditions. $64\pm0.55437$, approximately $[63.446,64.554]$. This is a large-sample mean interval, not a range containing $95\%$ of individual grades.

3.4.4 Two independent means

For iid samples within each group and independence between groups, target $\delta=\mu_X-\mu_Y$, estimated by $\hat\delta=\bar X-\bar Y$. Subtracting changes the sign of the centre, but independent variances still add:

The estimated expression is not the unknown true variance itself. No equality of population variances is assumed in this formula.

Anchoring: state the direction before subtracting

Use the lecture's illustrative anchoring exercise: $89$ students were split between a low prompt, $10$, and a high prompt, $65$, before giving a percentage estimate in a UN-membership question. Treat the reported summaries as teaching inputs: low group $n_X=43,\bar x=17.1,s_X^2=86$; high group $n_Y=46,\bar y=30.7,s_Y^2=253$. No current factual percentage is needed to analyse the responses.

1. Define the contrast. Low minus high is $17.1-30.7=-13.6$ percentage points.
2. Estimate its sampling variance. $86/43+253/46=2+5.5=7.5$. SE is $\sqrt{7.5}\approx2.73861$.
3. Construct the interval. Margin $1.96\sqrt{7.5}\approx5.36768$, giving $[-18.968,-8.232]$ percentage points.
4. Interpret the sign. The interval is entirely negative for low minus high. Reversing the contrast gives $[8.232,18.968]$, not the same signed interval.

Under the working model, the response difference is inconsistent with zero at the corresponding two-sided level. Random assignment can support a causal comparison of prompts, subject to implementation and response assumptions. It does not itself make these students representative of a wider population. Sampling and treatment assignment answer different questions.

3.4.5 Paired means: analyse the within-pair difference

If each student takes both examinations, scores within a student may be related. Define $D_i=X_i-Y_i$ for each student. Treat the $n$ pairs as independent sampling units; there are $n$ differences, not $2n$ independent observations.

Read $s_D$ as “s sub D,” the sample SD of differences. The covariance term follows by expanding $\operatorname{Var}(X-Y)$. Positive covariance reduces variation of differences: a student who scores high on both exams may still have a modest score difference. Marginal score SDs alone do not supply $s_D$.

For the lecture's $71$ students, the exam means are $79.6$ and $81.4$, variances $117$ and $151$, and sample correlation approximately $0.54$. The supplied difference variance is $124.45$, consistent with those rounded summaries. With exam 1 minus exam 2, the displayed example pairs $(57.1,60.7)$ and $(78.6,82.9)$ have differences $-3.6$ and $-4.3$.

1. Centre. $\bar d=79.6-81.4=-1.8$.
2. SE of the difference mean. $\sqrt{124.45/71}\approx1.32394$.
3. Margin and interval. $1.96\times1.32394\approx2.59492$, so the large-sample $95\%$ interval is $[-4.395,0.795]$.
4. Conclusion. The interval includes zero. These data do not establish a nonzero population mean score difference at this level. They do not prove equal means, and score differences alone cannot isolate exam difficulty from all other changes.

3.5.1 A proportion is a mean of indicators

Record $X_i=1$ when person $i$ gives the response of interest and $X_i=0$ otherwise. Then $p=P(X_i=1)$ is the population proportion, and $\hat p=n^{-1}\sum_iX_i=k/n$ is the sample proportion when $k$ successes are observed. Under iid Bernoulli sampling,

The large-sample Wald interval substitutes this estimated SE into a normal margin:

Read “p hat plus or minus c times its estimated standard error.” The population variance is determined by $p$, but $p$ is unknown, so the CI denominator uses $\hat p$. This will differ from a null-based test denominator. Applying the generic mean formula $s/\sqrt n$ to binary data instead gives $\sqrt{\hat p(1-\hat p)/(n-1)}$. The customary Wald formula uses the plug-in convention shown here; these two estimated SEs are asymptotically equivalent, and neither makes the interval exact.

A survey example with bounded assumptions

The lecture reports a historical survey example with $n=1502$ and an approval proportion $\hat p=0.39$. Use these as classroom inputs for an iid, unweighted approximation. A published survey may have weighting and a more complex design; these summaries alone do not establish its actual design-based standard error. The rounded percentage also does not determine an exact integer success count.

1. Variance of an individual indicator. Plug-in value $0.39(1-0.39)=0.2379$.
2. Variance and SE of the mean. Divide by $1502$, then take the square root: $\sqrt{0.2379/1502}\approx0.0125853$.
3. Margin. $1.96\times0.0125853\approx0.0246671$.
4. Interval. $[0.3653,0.4147]$, or approximately $36.53\%$ to $41.47\%$.

The margin is about $2.47$ percentage points, not a $2.47\%$ relative change in approval. Keeping probability and percentage scales explicit avoids a factor-of-$100$ error.

Why boundaries require a better procedure

When $\hat p=0$ or $1$, the Wald SE is zero. The resulting zero-width interval clearly does not express the uncertainty from a small sample. Wald endpoints can also lie outside $[0,1]$, and coverage may be poor near boundaries. Merely clipping the endpoints does not repair the coverage calculation. The score-test inversion below produces a Wilson interval that respects the probability scale; exact binomial procedures are another option.

3.5.2 Two independent proportions

For independent groups, estimate the difference $\delta=p_X-p_Y$ by $\hat\delta=\hat p_X-\hat p_Y$. The two estimated variances add because the samples are independent:

There is no reason to pool the two proportions for this interval: the target permits different population proportions. The equality test later imposes a common proportion under its null and consequently uses a different denominator.

For a historical lecture knowledge question about identifying John Roberts, take the reported inputs as illustrative: $239$ Republican respondents with correct-response proportion $0.47$, and $238$ Democrat respondents with proportion $0.31$. These rounded figures describe one specific question. They do not measure general political knowledge or prove one group is “better informed” overall.

1. Define the contrast. Republican minus Democrat correct-response proportion: $0.47-0.31=0.16$.
2. Unpooled SE. Compute the two variance contributions: $$\begin{aligned}a&=0.47(0.53)/239,\\ b&=0.31(0.69)/238,\\ e&=\sqrt{a+b}\approx0.044057.\end{aligned}$$
3. Interval. $0.16\pm1.96(0.044057)$, approximately $[0.07365,0.24635]$, or $7.37$ to $24.63$ percentage points.
4. Scope. Under the working independent-sample approximation the population contrast for this response is positive. Representativeness, wording, response and design affect any broader inference.

3.5.3 Paired proportions need the joint table

Before smoking is reported as $30\%$ and after smoking as $25\%$ for $100$ people. Those margins give the change $-0.05$, but not its SE. People who quit and people who start create the variance; people unchanged in either state create zero differences.

A coherent illustrative paired table

The following table is an explicitly constructed teaching example with those same marginal counts. It is not a recovered dataset from the lecture summary.

1. Compute the difference sum. Ten quitters contribute $-10$, five starters $+5$, and $85$ unchanged people zero. Therefore $\sum D_i=-5$ and $\bar d=-0.05$.
2. Compute squared differences. Each discordant pair contributes $1$, so $\sum D_i^2=15$. Use $\sum(D_i-\bar d)^2=\sum D_i^2-n\bar d^2=15-0.25=14.75$.
3. Estimate variance and SE. $s_D^2=14.75/99\approx0.14899$; $s_D/\sqrt{100}\approx0.03860$.
4. Construct the interval. $-0.05\pm1.96(0.03860)$, approximately $[-0.12565,0.02565]$.

The interval includes zero under this approximate procedure. That does not show that there is no change. It describes substantial uncertainty about the mean change. A before-and-after comparison without a suitable counterfactual also does not isolate the program's causal effect from other changes over time.

3.5.4 Practise choosing the denominator

3.6.1 State a claim about the population

A hypothesis concerns a population parameter, not the already observed statistic. “The population mean grade is $66$” is a testable claim $H_0:\mu=66$. “The observed sample mean is $64$” is a description of data, not an uncertain population hypothesis.

3.6.2 Error rates describe repeated decisions

A Type I error rejects a true null. A Type II error fails to reject a false null. The significance level $\alpha$ controls the rejection probability under the null:

At a specified alternative value $\theta$, the Type II probability is $\beta(\theta)$, read “beta at theta,” and power is $1-\beta(\theta)$: the probability of rejecting there. Power depends on effect size, variation, sample size and the rejection rule. Lowering $\alpha$ with everything else fixed usually makes rejection harder and reduces power.

1. If $H_0$ is true, $Z\sim N(0,1)\text{.}$ If the true mean lies $2.5$ standard errors above the null value, $Z\sim N(2.5,1)\text{.}$
2. At $5\%\text{,}$ reject when $|Z|>1.96\text{:}$ the red tails of the null curve have area $0.05\text{.}$ The power is the green area of the true curve beyond the cutoffs, $\Phi(0.54)\approx0.705\text{.}$
3. Lower the level to $1\%\text{:}$ the cutoffs move out to $\pm2.576\text{.}$ The red tails shrink to $0.01\text{,}$ and the power falls to $\Phi(-0.076)\approx0.470\text{.}$
4. The yellow strip between $1.96$ and $2.576$ is the power given up, about $0.235\text{.}$ A smaller Type I error bound costs power unless the sample size rises.

The trial analogy helps explain “fail to reject” versus “prove true”: an acquittal does not establish innocence. But the analogy does not make Type I errors universally worse. Error costs depend on the application, and statistical thresholds are not legal standards.

3.6.3 Build and calibrate the statistic

The test compares the observed distance from the null with the distance expected from sampling variation under that null. A common large-sample form is

The subscript $0$ stresses null calibration. The numerator is an effect in its original units; division expresses it in SE units. Under the assumptions of the relevant procedure, $Z$ is approximately $N(0,1)$ under $H_0$. A statistic far from zero is unusual under the whole null model, including its sampling and variance assumptions.

1. State the estimand and hypotheses. Fix the subtraction direction and whether the alternative is one- or two-sided.
2. Construct the statistic. Use the SE appropriate to the design and the null.
3. State the null distribution and assumptions. Distinguish exact calibration from an approximation.
4. Choose the level. For a two-sided normal test, reject when $|Z|>z_{1-\alpha/2}$; each tail has probability $\alpha/2$.
5. Report the decision in context. Give the estimated effect, uncertainty and evidence. Use “fail to reject,” not “accept” or “prove equal.”

Two-sided normal critical values are $1.645$ at $10\%$, $1.960$ at $5\%$, and $2.576$ at $1\%$. For a pre-specified upper-sided $5\%$ test, the critical value is $1.645$. The same numerical cutoff can therefore have different meanings depending on the tail rule.

3.6.4 A p-value is a null-model tail area

Definition. A p-value is the probability, calculated under the null model, of a test statistic at least as extreme as the observed statistic according to the pre-specified alternative.

For a two-sided standard normal test with observed value $z_{\rm obs}$,

Here $p_{\rm val}$ labels a p-value; it is unrelated to a population proportion $p$. For an upper-sided test use $1-\Phi(z_{\rm obs})$; for a lower-sided test use $\Phi(z_{\rm obs})$. A small p-value indicates that the statistic is unusual under the null model.

For a continuous calibrated statistic, the rule $p_{\rm val}<\alpha$ matches the critical-value rule. Discrete exact tests can have attainable rejection probabilities below the nominal level; precise equality conventions can matter at a boundary. We use non-rejection at equality in the displayed continuous rules.

3.6.5One test from start to finish

Return to the grade sample from 3.4: $n=100$ scripts, sample mean $\bar x=64$ and sample variance $s^2=8$. Suppose a department claims that the population mean grade is still $64.5$. Follow the five steps of 3.6.3.

1. If $H_0:\mu=64.5$ were true, means of $100$ scores would vary around $64.5$ with $\operatorname{SE}=\sqrt{8/100}\approx0.2828\text{.}$ The observed mean is $64\text{.}$
2. Standardise: $z=(64-64.5)/0.2828\approx-1.77\text{.}$ The observed mean lies $1.77$ standard errors below the claim.
3. Two-sided, a result at least as extreme lies $1.77$ or more standard errors away in either direction: $p=2\{1-\Phi(1.77)\}\approx0.077\text{.}$
4. At $5\%$ the cutoffs are $\pm1.96$ and $0.077>0.05\text{,}$ so we fail to reject $H_0\text{;}$ at $10\%\text{,}$ with cutoff $1.645\text{,}$ we would reject. The p-value is computed assuming $H_0\text{,}$ so it is not the probability that $H_0$ is true.

Two comparisons show what the decision depends on. At $\alpha=0.10$ the critical value is $1.645<1.77$, so the same data would reject: the level is part of the rule and must be fixed in advance. In 3.7.1 the same sample rejects the claim $\mu=65$, which lies $3.54$ standard errors from $\bar x$. The interval $[63.45,64.55]$ contains $64.5$ but not $65$; 3.8 shows why intervals and tests agree in this way.