Day 4 · Thursday

From hidden structure to decisions you must answer for

Three days of labelled data end here. Today we look for structure that no label announces, follow the same tools into modern AI, and then ask what a deployed system owes the people it scores.

By the end of today

• Group cases without labels, and judge the grouping without flattering it;
• Find the few directions that carry most of the variation, and read their loadings;
• Explain how embeddings connect this week's tools to modern AI;
• Audit a deployed system for fairness, monitoring, and governance.

What we cover

• slides 6–17Part I: grouping without labelsK-means on the 32 companies, and how to judge a grouping.
• slides 18–30Part II: directions that carry the variationPCA on the stocks, then on the US yield curve.
• slides 31–37Part III: the same tools, one level upEmbeddings, retrieval, and where modern AI sits.
• slides 38–59Part IV: responsibilityFairness, monitoring, governance, and the close of the week.

Slides 60–70 are optional appendix material for students who want the full algebra.

Lecture materials

Review questions

• A leaf predicts the training average of its region: Day 2's mean, restricted to one rectangle. A tiny leaf is memorisation in miniature.
• For $B$ trees of variance $\sigma^2$ with average correlation $\rho$, the average has variance $\rho\sigma^2+(1-\rho)\sigma^2/B$. More trees shrink only the second term; the correlation sets the floor. The lever is decorrelating the trees, not adding more of them.
• One validation set is one noisy draw. Five folds average that noise into a steadier judge.
• PR beats ROC when positives are rare, as with the bank file's 11.3% subscribers.
• The two audit questions: could every feature be known at prediction time, and does the split imitate deployment.

← Day 3  ·  All four days →