2.1.1What is being measured?

A binary indicator deserves special attention. Coding employed as $1$ and unemployed as $0$ makes its average the proportion employed. That interpretation follows from the chosen coding; an average of arbitrary eye-colour codes does not have an analogous meaning. Likewise, ordinal codes preserve ranking without establishing that the improvement from poor to fair equals the improvement from fair to good.

Measurement precision and a probability model are different choices. Wages recorded to the nearest penny are discrete records, but a continuous approximation may be useful. A rounded age is not proof that age itself is restricted to whole years. State which approximation is being used and what it is intended to describe.

2.1.2Support and several measurements

The support identifies values the model permits. A binary indicator has support $\{0,1\}$; a six-sided die has support $\{1,2,3,4,5,6\}$; a finite nonnegative lifetime can be modelled on $[0,\infty)$. Discrete supports need not be integers: a variable taking only $0.5$ and $1.5$ is discrete. A continuous support need not include every real number.

For a discrete distribution we usually list the values with positive probability. For a distribution with a density, support is described by the range where density can occur, including boundary points where appropriate. An observed sample range is not necessarily the population support: not observing an extreme value does not make it impossible.

A univariate measurement records one feature. A bivariate measurement records two, and a multivariate measurement records several. A household with five children whose youngest is four years old can be represented by the vector $\mathbf X=(5,4)$. The coordinates mean different things and must retain their labels. Height, weight and eye colour also form a multivariate record, even though the last coordinate is categorical.

A number-line plot can show one measurement per unit; a scatter plot can show two. Each point in a scatter plot represents a paired record from the same unit. Combining the height of one person with the weight of another would change the joint data and destroy information about their relationship. Three numerical features form points in three-dimensional space; more features remain meaningful even when they cannot be pictured directly.

2.1.3Cross-sections, time series and panels

A panel follows several units over time. Write $X_{it}$: the first index identifies a unit and the second an occasion. A balanced panel observes every unit on the same complete set of occasions. An unbalanced panel has a different observed time set for some units, perhaps because units enter or leave or measurements are missing. A panel therefore combines cross-sectional and temporal variation; it does not automatically eliminate dependence within a unit.

2.1.4Check the interpretation before calculating

2.2.1From records to probability masses

Variation can be described by assigning a probability to each possible measurement. For a discrete variable, a probability mass function, abbreviated PMF, gives that assignment. Discrete values may form a finite or countably infinite list. Probability belongs to each listed value; it is not determined by the space between values on a plot.

Consider the marks of ten scripts:

Select one script uniformly from this pile and let $X$ be its mark. “Uniformly” means each script has probability $1/10$. Three scripts have mark $30$, so the event $X=30$ contains three distinct selection outcomes and has probability $3/10$. Combining equal marks gives:

If the script is replaced and the pile shuffled before each new uniform draw, the same PMF applies each time and independent sampling can be modelled. A draw without replacement has this same first-draw PMF, but subsequent probabilities depend on what was removed. The pile’s relative frequencies exactly describe uniform selection from this pile; treating them as the distribution of a larger student population is a further inferential step.

2.2.2Bernoulli and categorical distributions

A Bernoulli variable records whether an event occurs: $X=1$ for success and $X=0$ otherwise. Fix one convention throughout: $p=P(X=1)$. Then

For $0<p<1$, both outcomes have positive probability, and one compact PMF covers them:

Setting $x=1$ leaves $p$; setting $x=0$ leaves $1-p$. The two-outcome formula above also handles $p=0$ or $p=1$ directly, without needing an endpoint convention for $0^0$.

For a fair coin, code heads as $1$; then $p=1/2$. For a hypothetical weighted coin with heads probability $9/10$, the masses at $0$ and $1$ are $0.1$ and $0.9$. “Success” is a label chosen for the event, not necessarily a desirable outcome. Changing that coding changes the parameter’s interpretation.

A categorical distribution, also called a generalized Bernoulli or multinoulli distribution, extends the idea to $k$ categories, with probabilities $p_1,\ldots,p_k$. Each $p_j$ is nonnegative and the probabilities sum to one. The categories can be named or associated with numerical values $x_1,\ldots,x_k$. The marks distribution is categorical over its five distinct marks, while the marks themselves retain numerical meaning.

The Iverson bracket converts a statement into a number: $\mathbf1\{x=x_j\}$ is one when the equality is true and zero otherwise. An alternative notation is $[x=x_j]$. With this indicator,

Only the term matching the selected value contributes. The formula is a compact version of the table, not a different probability model.

2.2.3New variables and summation notation

A transformation creates a new measurement $Y=g(X)$. To find its distribution, collect the probabilities of all original values producing the same new value:

Read the condition under the sum as “over the x values for which g of x equals y.” For an illustrative pass rule of at least $60$, define $Y=\mathbf1\{X\geq60\}$. Four distinct marks contribute to $Y=1$, so $P(Y=1)=0.2+0.3+0.1+0.1=0.7$. The mark distribution and pass distribution describe the same scripts at different resolutions.

For binary observations, the sum counts ones. Their average, $\bar X=N^{-1}\sum_{i=1}^{N}X_i$, is therefore the proportion of successes. Its possible values are $\{0,1/N,2/N,\ldots,1\}$. Although it can have many decimal values when $N$ is large, it remains discrete for fixed $N$. Infinite sums are also written with sigma notation, but a finite value requires the relevant series to converge.

2.2.4Counting successes: the binomial distribution

Suppose $N$ trials are independent, each has the same success probability $p$, and each produces a Bernoulli outcome $X_i$. The count $Y=\sum_{i=1}^{N}X_i$ has a binomial distribution, written $Y\sim\operatorname{Bin}(N,p)$. For $0<p<1$, its support is $\{0,1,\ldots,N\}$. At $p=0$ it is always zero, so the support reduces to $\{0\}$; at $p=1$ it is always $N$, with support $\{N\}$.

1. One ordered sequence. A particular sequence with $k$ successes and $N-k$ failures has probability $p^k(1-p)^{N-k}$, by independence.
2. Count sequences. There are $\binom Nk$ ways to choose the success positions. Read this as “N choose k.”
3. Add disjoint outcomes. The event of exactly $k$ successes includes every such sequence, so

The combinatorial multiplier depends on $N$ and $k$, rather than being one constant across all counts. Using $n!=n(n-1)\cdots1$, read “n factorial,” and $0!=1$, it is $\binom Nk=N!/[k!(N-k)!]$. The binomial theorem verifies that the masses sum to $(p+(1-p))^N=1$.

For three fair tosses and exactly two heads, the sequences are HHT, HTH and THH. Each has probability $1/8$; adding them gives $3/8$. Counting only one sequence would lose two thirds of the event. With $N=10$, $p=1/2$ and $k=5$, the probability is $252/1024=0.24609375$.

Increasing $N$ from $10$ to $15$ to $20$ expands the count’s support. For fixed $N=10$, compare $p=1/3,1/2,0.9$: the first places more mass near small counts with a right tail, the second is symmetric, and the third concentrates near ten with a left tail. At $p=1/2$, the distribution is symmetric and its variance is largest among binomials with that fixed $N$. Expectations and variances below will explain these shape changes algebraically.

2.2.5Worked checks on masses and multiplicity

2.3.1Probability per unit of measurement

A finite counting model assigns probability to separate values. For a variable with a continuous density, probability instead belongs to intervals. A probability density function, abbreviated PDF, tells us how much probability is concentrated per unit of measurement near a value.

Density requirements. A density is nonnegative and has total area one:

Read the integral as “integrate f sub X of x over all x.” An interval’s probability is its area under the curve:

The density value $f_X(x)$ is not itself a probability. It may exceed one without violating any probability rule. For a variable with a density, $P(X=x)=0$: a point has zero width and hence zero area. The probability of observing some real value is still one; probability is not obtained by summing point masses over an uncountable interval.

A density histogram illustrates the distinction. If a bin of width $w$ contains proportion $r$ of observations, its density height is $r/w$; its area remains $r$. Narrowing bins can reveal a distribution’s shape when enough observations are available. With a fixed small dataset, arbitrarily narrow bins do not automatically produce a smooth population density. The discrete ten-script mark distribution remains a PMF unless a different, continuous model is deliberately introduced.

2.3.2The uniform density: calculate with rectangles

For $X\sim U(a,b)$, read “X has a uniform distribution from a to b,” assume $a<b$. Equal-length intervals inside the support have equal probabilities. Its density is constant:

1. Width. The support has length $b-a$.
2. Normalization. A rectangle with height $c$ has total area $(b-a)c$. Set this to one.
3. Solve. The height must be $c=1/(b-a)$.
4. Restrict the interval. For $a\leq c<d\leq b$, its probability is $(d-c)/(b-a)$.

The symbols $c,d$ in the last step label interval endpoints, not the temporary height in the normalization step. More generally, clip an interval to the support before measuring its width. Parts outside $[a,b]$ carry no probability.

For the last distribution, density $2$ is compatible with probability one because its width is only $1/2$. The interval from $1.5$ to $1.75$ has width $1/4$, so its probability is $2(1/4)=1/2$.

1. $U(-1,1)\text{:}$ a rectangle of width $2$ and height $1/2$ has area $1\text{.}$
2. Squeeze the support to $(0,1)\text{:}$ the height doubles to $1$ and the area stays $1\text{.}$
3. Squeeze again to $(1.5,2)\text{:}$ height $2\text{,}$ area still $1\text{.}$ A height above $1$ is allowed, because it is a density, not a probability.
4. Zoom in: the probability of $(1.6,1.9]$ is its area, $0.3\times2=0.6\text{.}$
5. Shrink the interval to the point $1.7\text{:}$ its area, and so its probability, falls to $0$ while the height stays $2\text{.}$

2.3.3A normal family has location and spread

The normal distribution, also called Gaussian, is a family of bell-shaped densities. Write $X\sim N(\mu,\sigma^2)$, where $\mu$ is its mean and $\sigma>0$ its standard deviation. The second parameter in this notation is the variance, $\sigma^2$, not the standard deviation. Those measures are developed formally in the moments chapters.

Changing $\mu$ from $-1$ to $0$ to $2$, while keeping $\sigma=1$, shifts the curve without changing its spread. Keeping $\mu=0$ and changing $\sigma$ from $0.5$ to $1$ to $2$ produces increasingly wide, lower curves. Their areas remain one; a lower peak does not mean that the total probability is smaller.

A normal distribution’s mathematical support is the entire real line. A bell shape alone does not establish that a measurement is normal, and the model can be inappropriate when hard boundaries or heavy tails matter. Normal probabilities describe areas, not curve heights. The next chapter supplies the cumulative function needed to calculate them.

2.3.4Density, height and probability

2.4.1Accumulating probability up to a threshold

For the ten-script pile, the cumulative masses at marks $30,60,80,90,100$ are respectively $0.3,0.5,0.8,0.9,1$. Between marks, nothing new is added. Thus $F_X(29.999)=0$, $F_X(30)=0.3$, $F_X(30.01)=0.3$, and $F_X(70)=0.5$. The function is a staircase, not a smooth curve.

Every CDF lies between zero and one, is nondecreasing, approaches zero far to the left, and approaches one far to the right. It is right-continuous: its value at a jump includes the mass at that threshold. A discrete CDF need not be continuous. For a distribution with a density, the CDF is continuous, but need not be differentiable at every point, as a uniform distribution’s endpoints illustrate.

2.4.2Interval probabilities and endpoint choices

An interval probability subtracts the mass already accumulated before the interval from the mass accumulated through its upper endpoint:

The first expression excludes $a$, because its mass is included in the subtracted $F_X(a)$. For discrete variables, changing an endpoint from strict to inclusive can change the answer. Write $F_X(a^-)$ for the CDF limit just below $a$, read “F sub X at a from the left.” Then

For the marks, $P(30<X\leq80)=0.8-0.3=0.5$, whereas $P(30\leq X\leq80)=0.8-0=0.8$. The difference is the three scripts marked $30$. For a variable with a density, endpoint masses are zero, so these strict and inclusive versions coincide.

For $X\sim U(a,b)$, accumulating the rectangle’s area gives

For $U(0,2)$, $F_X(0.8)=0.4$ and $F_X(1.5)=0.75$; their difference is $0.35$. The same area can be found directly from width times height. Where the CDF is differentiable, its slope is the density: $f_X(x)=F_X'(x)$. A steep CDF means probability accumulates quickly in that region, not that cumulative probability can exceed one.

2.4.3Reverse the question: quantiles

A probability calculation starts with a threshold and finds a probability. A quantile starts with a probability and finds a threshold. The $p$-quantile, for $0<p<1$, can be defined by

Read $\inf$ as “infimum,” the lowest threshold satisfying the condition. For a continuous strictly increasing CDF this is the inverse function and solves $F_X(x)=p$. The inequality definition also works at jumps, where no threshold may have exactly the requested cumulative probability.

The median corresponds to $p=0.5$; quartiles use $0.25,0.5,0.75$; deciles use tenths; percentiles use hundredths. In a CDF plot, start at the requested probability on the vertical axis, move to the curve, then read the threshold on the horizontal axis. For a uniform distribution, solve the linear CDF:

For the marks, the lower-quantile convention gives $Q_X(0.25)=30$, $Q_X(0.5)=60$, and $Q_X(0.75)=80$. A conventional sample median averages the fifth and sixth ordered records, giving $(60+80)/2=70$. These are different conventions for a distribution with a gap at its halfway point. Every value between $60$ and $80$ satisfies the usual median inequalities; specify the convention when a unique numerical answer is needed.

As a continuous example, a normal model $N(5,4)$ has its $10$th percentile near $2.44$, first quartile near $3.65$, median $5$, third quartile near $6.35$, and $90$th percentile near $7.56$. Standardisation will show how these numbers are obtained from one common reference distribution.

2.4.4Normal areas and table reading

For a standard normal variable $Z\sim N(0,1)$, write its CDF as $\Phi(z)$, pronounced “capital phi of z.” It is an S-shaped cumulative curve, while its density is bell-shaped. Normal areas have no elementary antiderivative; numerical routines or probability tables calculate them.

In a two-dimensional table, a row such as $1.9$ and a column such as $0.06$ identify $z=1.96$. A negative-value table may use row $-2.3$ and column $0.03$ to identify $-2.33$; inspect its sign convention. The entry is a left-tail probability, not a density height. At $-2.33$, that probability is about one percent, $0.0099$, rather than $0.099$.

1. Left tail. $P(Z\leq-1.02)=\Phi(-1.02)=0.1539$.
2. Right tail. $P(Z\geq1)=1-\Phi(1)=0.1587$. Normal symmetry also gives $\Phi(-1)=1-\Phi(1)$.
3. Interval. For $-0.5\leq Z\leq2$, subtract the left tails:

To find the third quartile from the table, $0.75$ lies between $0.7486$ at $0.67$ and $0.7517$ at $0.68$. Linear interpolation uses the fraction of the probability gap traversed:

A more precise numerical inverse gives $0.674490$. Interpolation is an approximation between rounded table entries, not a claim that the normal CDF is exactly linear there.

A central $95\%$ interval leaves $2.5\%$ in each tail. Its upper endpoint is therefore the $97.5$th percentile, not the $95$th. In particular $P(|Z|\geq1.645)\approx0.10$, while $\pm2.5758$ describes a central $99\%$ interval. These probabilities belong to the specified normal model.

2.4.5Work from the event to its cumulative probabilities

2.5.1The mean is a probability-weighted balance

A distribution records the full allocation of probability. An expected value summarizes its location by weighting each value according to how much probability it receives. “Expectation,” “mean” and “expected value” refer to the same mathematical object. Expectation does not mean the outcome that will occur, or necessarily the outcome that occurs most often.

For the ten-script pile, averaging individual marks and averaging distinct marks with their probabilities give the same result:

The individual records sum to $640$, so their ordinary mean is also $640/10=64$. Grouping repeated marks changes the calculation’s organization, not its result. The mean is the balance point of the probability weights: $E[X-\mu]=0$.

No script has mark $64$, yet it is the expected mark of a uniform selection. A fair die likewise has mean $(1+2+3+4+5+6)/6=3.5$, which is not a possible face. The modes of the mark distribution are $30$ and $80$, each with probability $0.3$; neither equals the mean. These examples separate an average from a prediction of one exact outcome.

2.5.2Indicators and continuous averages

For a Bernoulli indicator with $P(X=1)=p$,

Its expectation is the probability of the event coded one. For the blue-eye indicator record $1,0,0,1,1,0,1,0,0,0$, four of ten entries are one. Its mean is $0.4$, the proportion coded blue-eyed. This is an average of indicators, not an average eye colour.

For a uniform variable, the mean is its midpoint. Derive it by weighting each value with the constant density:

Thus $U(10,20)$ has mean $15$. For $N(\mu,\sigma^2)$, the mean is $\mu$. Its density is symmetric about $\mu$, so equal probability weights at opposite distances cancel in the expectation of $X-\mu$. This argument relies on the mean existing; symmetry alone is not sufficient.

2.5.3Transform values first, then average

For a function $g$, the expectation of the transformed measurement is found using the original probability weights:

The first expression is for discrete values and the second for a density. This method avoids constructing the transformed distribution when only its average is required. It also makes clear that averaging and transforming are two separate operations.

For squared marks,

Squaring the mean instead gives $64^2=4096$. The difference comes from variation around the mean. It will become the variance in the next chapter.

Linearity. For constants $a,b$ and integrable variables,

1. Expand the weighted sum. Each term of $E[a+bX]$ is $(a+bx)p_X(x)$.
2. Separate terms. Sum the constant part and the variable part separately.
3. Normalize. The constant part is $a\sum_xp_X(x)=a$; the remaining part is $bE[X]$.

The same separation works for integrals and for sums of variables. Addition does not require independence. Multiplication is different: independence gives $E[XY]=E[X]E[Y]$, when the required expectations exist. Without that condition, the joint distribution determines the average product. More generally, independent variables also allow $E[g(X)h(Y)]=E[g(X)]E[h(Y)]$ under integrability.

For $N$ binary trials with common success probability $p$, linearity gives $E[\sum_iX_i]=Np$ and $E[\bar X]=p$. Independence is unnecessary for these two mean identities; it is needed for the binomial distribution of the count.

2.5.4Nonlinearity and the existence of moments

A convex function bends upward: a straight segment joining two points on its graph lies on or above the graph. A concave function reverses that geometry. Jensen’s inequality, pronounced “YEN-sen’s inequality,” extends that comparison to probability-weighted averages, assuming the relevant expectations exist:

For convex $g$,

For concave $g$,

Squaring and exponentiation are convex; logarithms on positive values and square roots on nonnegative values are concave. Consequently $E[X^2]\geq(E[X])^2$, $E[e^X]\geq e^{E[X]}$, $E[\log X]\leq\log E[X]$ for positive $X$, and $E[\sqrt X]\leq\sqrt{E[X]}$ for nonnegative $X$. Equality can occur, for example when there is no variation; “nonlinear” does not automatically mean every instance is strictly unequal.

Ratios also need care. Suppose two paired outcomes $(X,Y)=(1,1)$ and $(1,2)$ are equally likely. Then $E[X/Y]=(1+1/2)/2=3/4$, while $E[X]/E[Y]=1/(3/2)=2/3$. Averaging ratios is not generally the same as taking the ratio of averages.

2.5.5Choose the correct order of operations

2.6.1Why square deviations?

Two distributions can have the same mean and very different spread. Ordinary deviations cannot summarize spread by their average, because positive and negative deviations from the mean cancel: $E[X-\mu]=0$. Squaring prevents that cancellation, weights larger deviations more strongly, and produces a nonnegative summary.

Definition. With finite second moment and $\mu=E[X]$,

Read “variance of X.” This is the second central moment. The raw second moment is $E[X^2]$; it is related to variance but is not the same object.

1. Expand the square. $(X-\mu)^2=X^2-2\mu X+\mu^2$.
2. Apply linearity. The fixed $\mu$ can be taken outside the expectation.
3. Substitute the mean. Since $E[X]=\mu$,

For a PMF, variance is $\sum_x(x-\mu)^2p_X(x)$; for a density, it is $\int(x-\mu)^2f_X(x)\,dx$. Its nonnegativity follows directly from averaging squared deviations. A finite-variance variable has variance zero exactly when it equals a constant with probability one.

2.6.2The same marks, now measuring spread

For the mark distribution, $\mu=64$, $E[X^2]=4720$, and $\mu^2=4096$. Therefore