The learning route

These five topics form a connected course in probability and statistics. Each changes the question we ask of data: what can happen, how measurements vary, how samples inform a population, what time dependence changes, and which comparison identifies an effect.

Each topic has a lecture: present its slides in the browser, or read the same slides as a PDF with every step shown. Within each topic, read chapters in order. Topic 2 places expectation and variance before normal standardization so that its subtraction and division have a clear meaning. Topic 3 treats single, independent and paired designs explicitly. The later topics revisit the assumptions behind earlier formulas.

How to study a derivation

1. Name the object. State the variable, event, population or causal target before using a formula.
2. Read the assumptions. Equal weights, independence, finite moments, stationarity and conditional independence serve different purposes.
3. Follow each step. The notes distinguish definitions, algebraic identities and results that need assumptions.
4. Predict, then inspect. Use a control when it changes a denominator, weighting scheme, sampling distribution or model. Explain why the result moves.
5. Rework the exercises. Attempt the question before opening its solution. Check both the numerical result and its interpretation.

The contents menu opens topics and chapters directly. All notes gives a continuous reading view. Each topic has a static print view with solutions expanded; the PDF contains the complete course. Mathematical notation is typeset locally, and the static notes remain readable without JavaScript.

1.1.1 Outcomes, events and a probability model

Before calculating: Name the experiment, describe one outcome, and identify all outcomes that make the event true. Then ask what probability weights the model supplies.

1.1.2 Three sample spaces

A die: fairness supplies the probability weights

A coin: an event space is a set of sets

Model one toss as having two recorded outcomes: $H$ for heads and $T$ for tails. This model excludes other recorded results. Then

The four events are neither outcome, heads, tails, and either outcome. The symbol $\varnothing$, read “the empty set”, denotes an impossible event; the whole sample space is a certain event. Notice the two levels of braces: $\{H\}$ contains an outcome, while $\mathcal F$ contains events. Each of the two outcomes can be included or excluded, giving $2^2=4$ events.

Enumerating events requires no fairness assumption. We could assign heads any probability $p$ between $0$ and $1$, then assign tails $1-p$. Here $p$, read “pee”, is a model parameter. The probability of heads becomes $1/2$ only when fairness supplies that value.

A bulb: intervals replace lists of outcomes

Separate two questions: Is a given lifetime in the interval? How much probability does the model assign to that interval? The first follows from the inequalities; the second needs a distribution.

1.1.3 “Not”, “and” and “or”

Before adding probabilities: List the distinct included outcomes. Does any outcome satisfy both statements? “Even or prime” includes the face $2$ once, although it answers both questions.

For a separate pair of events, suppose $P(A)=0.25$, $P(B)=0.15$, and $P(A\cap B)=0.05$. Their four disjoint regions have probabilities $0.20$ for $A$ only, $0.05$ for both, $0.10$ for $B$ only, and $0.65$ for neither. Their total is $1$. We obtain these values by keeping the overlap in its own region, then subtracting it from each marginal; Section 1.1.5 justifies that accounting.

In a quantitative diagram, area or width is deliberately scaled to probability. A unit square can represent the whole model, with total probability $1$; a region occupying one quarter then represents probability $0.25$. An ordinary Venn diagram need only represent membership and overlap: its drawn areas need not equal probabilities. Read the diagram’s scale before interpreting its geometry.

1.1.4 The axioms of probability

The axioms impose consistency on probability assignments. They prohibit negative mass, fix the total, and specify how distinct parts combine. They do not choose the weights of a particular die or lifetime model.

Axiom 1 — nonnegativity. For every allowed event, $P(A)\geq0$.

Axiom 2 — normalisation. $P(\Omega)=1$.

For example, two disjoint events with probabilities $0.2$ and $0.1$ have union probability $0.3$. Disjoint events with masses $0.1$ and $0.01$ similarly have union probability $0.11$. These calculations use disjointness, which prevents double counting; they require no equal-outcome assumption.

One immediate consequence is $P(\varnothing)=0$. The disjoint union $\Omega\cup\varnothing$ is still $\Omega$. Additivity therefore gives $P(\Omega)=P(\Omega)+P(\varnothing)$; subtracting $P(\Omega)=1$ leaves zero.

The converse needs care: an event with probability zero need not be the empty set. Continuous models can assign zero probability to a single exact value. The empty set is logically impossible; a zero-probability event is a statement about the model's mass assignment.

We now have a consistent way to add disjoint pieces. The next step is to decompose more complicated events into such pieces, then use the same axiom to derive useful rules.

1.1.5 Four rules, derived from the axioms

Complement rule

Goal: Express the probability outside $A$ using $P(A)$.

Proposition. $P(A^c)=1-P(A)$.

1. $A$ takes $0.25$ of the area. Everything else in $\Omega$ is $A^c\text{.}$
2. Together they make $\Omega\text{,}$ so $P(A\cup A^c)=P(\Omega)=1$ by Axiom 2.
3. They do not overlap, so $P(A\cup A^c)=P(A)+P(A^c)$ by Axiom 3.
4. Equate the two expressions: $P(A)+P(A^c)=1\text{.}$
5. Rearrange: $P(A^c)=1-P(A)=1-0.25=0.75\text{.}$

1. Set identity. Every outcome is in $A$ or outside it, so $A\cup A^c=\Omega$ and $A\cap A^c=\varnothing$.
2. Disjoint additivity and normalisation. $P(A)+P(A^c)=P(\Omega)=1$.
3. Rearrangement. Subtract $P(A)$ from both sides.

If an event has probability $0.25$, its complement has probability $1-0.25=0.75$. Also, nonnegativity of the complement gives $P(A)\leq1$. Together with Axiom 1, this proves the familiar bound $0\leq P(A)\leq1$.

The rule is useful when “not” is easier to describe than the desired event. For a bulb, “lasts more than $100$ hours” is the complement of “lasts at most $100$ hours.” The endpoint belongs to the latter event, so the two events are disjoint and exhaustive.

Union rule: remove the duplicated overlap

Proposition. $P(A\cup B)=P(A)+P(B)-P(A\cap B)$.

Adding the two marginal probabilities counts their intersection twice. Here “marginal” means the probability of each event considered separately, regardless of whether the other occurs. The derivation makes the required subtraction precise.

1. $\Omega$ has area $1\text{.}$ The events $A$ and $B$ are drawn to scale.
2. Paint $A\text{:}$ $P(A)=0.25$ includes the overlap.
3. Paint $B\text{:}$ the overlap is now painted twice, so $0.25+0.15=0.40$ is too large.
4. Lift off the second copy of the overlap. It becomes the subtracted term, $-P(A\cap B)=-0.05\text{.}$
5. Outline the union: each point is counted once, so $P(A\cup B)=0.35\text{.}$

1. Disjoint decomposition of $A$. $A=(A\setminus B)\cup(A\cap B)$. Additivity gives $P(A)=P(A\setminus B)+P(A\cap B)$.
2. Rearrangement. $P(A\setminus B)=P(A)-P(A\cap B)$.
3. Disjoint decomposition of the union. $A\cup B=(A\setminus B)\cup B$. These parts are disjoint, so additivity gives $P(A\cup B)=P(A\setminus B)+P(B)$.
4. Substitution. Insert Step 2 into Step 3 to obtain the proposition.

With the probabilities from Section 1.1.3, the rule gives $0.25+0.15-0.05=0.35$. Adding the three included regions, $0.20+0.05+0.10$, gives the same answer. The overlap is subtracted once because the desired union must still include it once.

Union bound

Proposition. $P(A\cup B)\leq P(A)+P(B)$.

Union rule: the union is the sum minus $P(A\cap B)$. Nonnegativity: this subtracted quantity is at least zero. Subtraction cannot increase the sum, which proves the bound. Equality holds when the overlap has probability zero; disjoint events guarantee that condition. The bound may exceed $1$, so $\min\{1,P(A)+P(B)\}$ is a tighter upper bound when necessary. Here $\min$ means “the smaller of”.

Logical consequence, or monotonicity

Proposition. If $B\subseteq A$, then $P(B)\leq P(A)$.

1. Set inclusion. If $B$ occurs, $A$ must occur: every outcome of $B$ is in $A$.
2. Disjoint decomposition and additivity. $A=B\cup(A\setminus B)$, hence $P(A)=P(B)+P(A\setminus B)$.
3. Nonnegativity. $P(A\setminus B)\geq0$, so $P(A)\geq P(B)$.

Adding a requirement restricts an event to a subset and cannot increase its probability. For the fair die, “six” implies “even”. In Section 1.1.2 these were $A=\{6\}$ and $B=\{2,4,6\}$, so that inclusion gives $P(A)=1/6\leq P(B)=1/2$. The letters do not determine which set is larger; identify the actual inclusion before applying the rule.

The same relationship holds for the hypothetical loaded die: $0.5\leq0.7$. Its numerical probabilities change, but the logical implication does not. Set relationships constrain every valid probability assignment.

1.1.6 Exercises with worked reasoning

For each question, identify the outcomes first. State the probability assumption, choose a rule, and explain why its conditions hold. Distinguish an exact answer from a bound: knowing that an event lies inside another can restrict its probability without determining it.

These rules combine probabilities within a fixed model. The next chapter considers what changes when we learn that a particular event has occurred.

1.2.1Information changes the question

A probability describes uncertainty relative to the information available. Before learning anything else, we consider the whole sample space. After learning that an event occurred, we consider only possibilities consistent with that event. Conditional probability expresses this change of reference group: among the possibilities compatible with what we know, how much probability belongs to the event we are asking about?

Consider a family with two children. Use a toy model in which each child has a boy or girl outcome, each with probability $1/2$. Assume the two outcomes are independent: knowing one does not change the probabilities for the other. Order the children from older to younger, writing $b$ for boy and $g$ for girl. These assumptions define the exercise; they are not demographic claims. The sample space is

$$\Omega=\{(b,b),(b,g),(g,b),(g,g)\}.$$

Independence gives each ordered pair probability $(1/2)(1/2)=1/4$. The mixed pairs are distinct: $(b,g)$ has an older boy, while $(g,b)$ has an older girl. Let $A$ be “two boys” and $B$ be “at least one boy.” Then

$$\begin{aligned}A&=\{(b,b)\},\\ B&=\{(b,b),(b,g),(g,b)\}.\end{aligned}$$

1. Four equally likely families: each has probability $1/4\text{.}$
2. Two boys is one of them: $P(A)=1/4\text{.}$
3. Learn $B\text{,}$ that there is at least one boy: $(g,g)$ is ruled out.
4. The three that remain share the whole, a third each, so $P(A\mid B)=(1/4)/(3/4)=1/3\text{.}$
5. No family changed; only the group we compare with did.

Specify how the information is obtained. Choose a family at random from this model, check whether it has at least one boy, and retain it exactly when $B$ is true. This procedure excludes $(g,g)$ without favoring one of the three retained outcomes over another. Of the retained probability $3/4$, the part belonging to two boys is $1/4$. Therefore

$$P(A\mid B)=\frac{1/4}{3/4}=\frac13.$$

Before screening, the probability of two boys was $1/4$; within the screened group it is $1/3$. No family has changed. We have changed the group within which the fraction is calculated.

Different information procedures give different answers

If we instead learn that the older child is a boy, only $(b,b)$ and $(b,g)$ remain. They are equally likely, so the probability of two boys is $1/2$. “The older child is a boy” and “at least one child is a boy” are different events, even though both imply that the family has a boy.

Now choose one of the two children at random, independently of the family’s outcomes, and observe that the selected child is a boy. A two-boy family always produces that observation; a mixed family produces it only half the time. Mixed families therefore receive less weight than two-boy families among observations of a boy. To count explicitly, pair each of the four family outcomes with either choice of child. There are eight equally likely family–child pairs. Four produce a boy: two from $(b,b)$, one from $(b,g)$, and one from $(g,b)$. Two of those four come from a two-boy family, giving $2/4=1/2$.

The statement we heard is not enough to justify a calculation if its reporting procedure is unspecified. In particular, a parent choosing which child to mention need not reproduce either procedure above. First define what was checked or observed; then identify the outcomes and weights consistent with that information.

1.2.2Restriction, normalization and the definition

Counting retained outcomes worked above because the relevant cases were equally likely. The general definition uses their probability weights instead. Let $A$ be the event we want to assess and $B$ the information on which we condition.

1. The model: $P(A)=0.2\text{,}$ $P(B)=0.6$ and $P(A\cap B)=0.1\text{.}$
2. Learn that $B$ happened: everything outside $B$ is ruled out.
3. $B$ swells to fill the square. Every part of it grows by the same factor, $1/0.6\text{,}$ so the overlap keeps its share of $B\text{:}$ $1/6\text{.}$
4. So $P(A\mid B)=P(A\cap B)/P(B)=0.1/0.6=1/6\text{,}$ and the two parts of $B$ still add to $1\text{.}$
5. Shrinking back multiplies by $P(B)\text{:}$ $P(A\cap B)=P(A\mid B)\,P(B)=1/6\times0.6=0.1\text{.}$

This process is called renormalization: restoring a total of one after restriction. For example, suppose $P(A)=0.2$, $P(B)=0.6$ and $P(A\cap B)=0.1$. The intersection contributes probability $0.1$, and the rest of $B$ contributes $0.6-0.1=0.5$. Write $A^c$, read “A complement,” for not $A$. Conditioning scales these two retained parts by $1/0.6$:

The rescaled parts add to one. The original probability $P(B)$ remains $0.6$; the new probability $P(B\mid B)$ is one. Conditioning creates a different probability assessment rather than overwriting the original one.

The product rule reverses the rescaling

If $P(A\mid B)$ describes the fraction inside $B$, multiplying by the size of $B$ restores its share of the original whole:

Product rule. When $P(B)>0$,

1. Conditional definition. Begin with $P(A\mid B)=P(A\cap B)/P(B)$.
2. Multiply both sides by the denominator. This gives the displayed product rule.

The left side is a joint probability: the probability that both events occur. It equals the probability of belonging to $B$, multiplied by the fraction of that group also in $A$. Replacing $P(A\mid B)$ with $P(A)$ would be an additional claim that the information $B$ leaves the probability of $A$ unchanged. The next chapter identifies that condition as independence.

The probability rules continue to hold

Keep $B$ fixed with $P(B)>0$. Nonnegativity and the fact that $A\cap B$ is contained in $B$ give

The reference group is certain under its own condition, while its complement is excluded:

For events $A_1,A_2,\ldots$ that are mutually exclusive within $B$, conditional probabilities add. “Within $B$” means that no outcome in $B$ belongs to two of these events, even if they overlap outside $B$.

The proof uses the earlier additivity rule on the disjoint intersections with $B$, then divides every term by the same $P(B)$. The notation $\bigcup_i A_i$ means “the union of the A sub i events,” and $\sum_i$ means “sum over i”; the list may be finite or countably infinite. In particular, the complementary parts inside $B$ satisfy

1.2.3Reversing the condition changes the denominator

“Among cases in $B$, how many are in $A$?” asks for $P(A\mid B)$. “Among cases in $A$, how many are in $B$?” asks for $P(B\mid A)$. These are different questions about the same joint event. Provided both conditioning probabilities are positive,

$$\begin{aligned}P(A\mid B)&=\frac{P(A\cap B)}{P(B)},\\ P(B\mid A)&=\frac{P(A\cap B)}{P(A)}.\end{aligned}$$

The intersection is symmetric: “$A$ and $B$” describes the same outcomes as “$B$ and $A$.” The conditional calculation is not symmetric, because it divides by a different reference group. Interchanging the letters therefore does not justify keeping the same value.

Represent the preceding probabilities using a hypothetical group of $100$ cases:

Four disjoint groups with the stated probability weights

Event	$B$	$B^c$	Total
$A$	$10$	$10$	$20$
$A^c$	$50$	$30$	$80$
Total	$60$	$40$	$100$

1. A hypothetical group of $100$ cases: $20$ in $A\text{,}$ $60$ in $B$ and $10$ in both.
2. Among the $60$ cases in $B\text{,}$ $10$ are in $A\text{:}$ $P(A\mid B)=10/60=1/6\text{.}$
3. Gather the $20$ cases in $A\text{:}$ half of them, $10\text{,}$ are in $B\text{,}$ so $P(B\mid A)=10/20=1/2\text{.}$
4. The same $10$ cases, compared with different groups, give different answers.

Conditioning on $B$ selects the column containing $60$ cases. Its $10$ cases in $A$ give $P(A\mid B)=10/60=1/6$. Conditioning on $A$ selects the row containing $20$ cases. Its $10$ cases in $B$ give $P(B\mid A)=10/20=1/2$. The $10$ cases in both events have not changed; the group used to interpret them has.

A verbal example makes the distinction clear. Suppose most students in a hypothetical population are Democrats. That need not mean that most Democrats in that population are students: the Democrat group may be much larger and include many nonstudents. One fraction uses the student total; the other uses the Democrat total. A larger denominator makes the fraction smaller for the same overlap, but group sizes and overlap are both needed to calculate how small.

Reverse conditionals can coincide in particular models. For example, if the two reference groups have the same positive probability, the equal numerators and denominators give equal answers. Equality in one example is not a general rule. Bayes’ rule in the next chapter connects the two directions while accounting for their different denominators.

1.2.4Recovering an overall probability from groups

Conditioning isolates a group. Often we need the reverse task: combine the probabilities within several groups to recover a probability for the whole population. This requires knowing both each group’s internal rate and its share of the population.

1. Split $\Omega$ into two groups, $B$ and $B^c\text{,}$ of sizes $0.6$ and $0.4\text{:}$ every case is in exactly one.
2. Inside each group $A$ has its own rate: $P(A\mid B)=1/6$ and $P(A\mid B^c)=1/4\text{.}$
3. Each rate times its group size is an area: $1/6\times0.6+1/4\times0.4=0.1+0.1=0.2\text{.}$
4. Averaging the two rates would treat the groups as equal and give about $0.208\text{.}$ The group sizes are the weights.

Why the group probabilities are weights

The rate $P(A\mid B_i)$ is measured within group $B_i$, where that group is treated as the whole. Multiplying by $P(B_i)$ converts it into a share of the original population. For an exhaustive, disjoint partition, these weights add to one. A large group contributes more than a small group with the same internal rate; simply averaging the rates would implicitly assign equal population shares.

Return to the table above. The groups $B$ and $B^c$ contain respectively $60\%$ and $40\%$ of cases. The probability of $A$ within $B$ is $10/60=1/6$; within $B^c$, it is $10/40=1/4$. Weight these rates by the corresponding group probabilities:

This recovers the original $20$ cases in $A$ out of $100$. The two contributions are equal even though the group sizes differ: the smaller group has the higher internal rate. If the groups overlapped, adding their contributions would count some outcomes twice. If they omitted possibilities, the result would leave out some probability.

Total probability accounts for every route to an event. That will make it useful in Bayes’ rule: before asking which group an observed event came from, we must calculate how often the event occurs across all groups.

1.2.5Exercises with worked solutions

Identify the conditioning group or partition before calculating. Try each problem before opening its solution.

1.3.1 Independence: information that leaves a probability unchanged

Proposition. If $P(B)>0$, the product definition and the unchanged-conditional definition are equivalent.

1. Conditional definition. Start with $P(A\mid B)=P(A\cap B)/P(B)$. The positivity condition permits division.
2. Product implies unchanged conditional. Substitute independence: $$\begin{aligned}P(A\mid B)&=\frac{P(A)P(B)}{P(B)}\\ &=P(A).\end{aligned}$$
3. Unchanged conditional implies product. Starting instead from $P(A\mid B)=P(A)$, multiply both sides by $P(B)$: $$P(A\cap B)=P(A)P(B).$$
4. Symmetry. Intersection and multiplication are commutative, so the product equation also gives $B\perp\!\!\!\perp A$. If $P(A)>0$, dividing by $P(A)$ gives $P(B\mid A)=P(B)$.

The product definition remains meaningful when a marginal is zero. If $P(A)=0$, monotonicity gives $P(A\cap B)=0$, so $A$ is independent of every $B$. The fraction $P(B\mid A)$ is then undefined in our elementary conditional-probability definition. Do not divide by zero to express the symmetry.

Independence also describes the information supplied by non-occurrence. Subtract the joint mass from the marginal: $P(A\cap B^c)=P(A)-P(A\cap B)$. Under independence this becomes $P(A)[1-P(B)]=P(A)P(B^c)$. If $P(B^c)>0$, division therefore gives $P(A\mid B^c)=P(A)$. Neither learning that B occurred nor learning that it did not occur changes the probability of A.

How much overlap means independence?

Consider $P(A)=0.30$, $P(B)=0.40$, and $P(A\cap B)=0.12$. The independence target is $0.30(0.40)=0.12$, exactly the stated overlap. Consequently, $P(A\mid B)=0.12/0.40=0.30$, and $P(B\mid A)=0.12/0.30=0.40$. The four disjoint region masses are $0.12$ for both, $0.18$ for A only, $0.28$ for B only, and $0.42$ for neither. They sum to one.

1. $B$ is the bottom band, $P(B)=0.4\text{.}$ $A$ takes $0.3$ of the width in both bands, so $P(A)=0.3$ and the overlap is $0.12\text{.}$
2. Learn $B\text{:}$ $B$ swells to fill the square and $A$’s share of it is still $0.12/0.4=0.3\text{.}$ Learning $B$ tells us nothing about $A\text{.}$
3. If more of $A$ lay inside $B\text{,}$ learning $B$ would make $A$ more likely: $P(A\mid B)=0.5\text{.}$
4. If $A$ and $B$ did not overlap, learning $B$ would rule $A$ out: $P(A\mid B)=0\text{.}$ Disjoint events are dependent.
5. Only when the shares match is $A$ independent of $B\text{:}$ $P(A\cap B)=P(A)\,P(B)=0.3\times0.4=0.12\text{.}$

Within B, the share belonging to A is the same as in the whole sample space. If the overlap increased while the two marginals stayed fixed, knowing B would increase the probability of A. If the overlap decreased, knowing B would decrease it. The product gives the point at which those probabilities match.

Disjointness answers a different question

Think first. If two positive-probability events cannot occur together, does learning that one occurred tell you anything about the other?

Disjoint events cannot occur together: $A\cap B=\varnothing$. If both marginals are positive, their joint probability is zero while their product is positive, so they are dependent. Knowing that $B$ occurred rules out $A$: its conditional probability becomes zero. Disjointness is about which outcomes can occur together; independence is about whether information changes a probability.

Disjoint events with a zero marginal can be independent; the positive-probability qualification matters. Statistical dependence, in turn, does not by itself establish a causal effect.

1.3.2 Bayes’ rule: the same joint probability in two directions

1. The model of chapter 1.2: $P(A)=0.2\text{,}$ $P(B)=0.6$ and $P(A\cap B)=0.1\text{.}$
2. From $B$’s side the overlap is $1/6$ of $B\text{:}$ $P(A\cap B)=P(A\mid B)\,P(B)=1/6\times0.6\text{.}$
3. From $A$’s side it is half of $A\text{:}$ $P(A\cap B)=P(B\mid A)\,P(A)=1/2\times0.2\text{.}$
4. Both describe the same area, so $P(A\mid B)\,P(B)=P(B\mid A)\,P(A)\text{.}$
5. Divide by $P(B)\text{:}$ Bayes’ rule, $P(A\mid B)=P(B\mid A)\,P(A)/P(B)=(1/2\times0.2)/0.6=1/6\text{.}$