Datasets and notebooks
Datasets and notebooks
Every file the course uses, listed by session. Each session has its notebook, the datasets that notebook reads, and a link to the page that documents each dataset in full.
data folder beside the notebook, open the
notebook in Jupyter, and run the cells in order. Every notebook uses the
relative path data/<file>, so nothing needs editing.Follows the mathematics workshop in order. The eight stores from the slides are the running example, so every formula is checked against an answer the deck already prints. London data enters twice, where eight rows are not enough: to show a sampling distribution, and to read $R^2$ out of sample.
Fits a rule by hand on four properties, so nothing about fitting stays hidden inside a library call. Then asks the harder question of the bank campaign: what evidence would let you claim the rule works on clients you have never seen?
Opens both models. Least squares on London prices, then the same grammar for a probability: the sigmoid, log loss, calibration, and a decision cut-off taken from the prices of the two mistakes rather than from a software default.
Fits models that can bend, and builds the testing discipline that stops them flattering themselves. It ends with two evaluations that reverse each other's verdict on the same data.
The answer column is taken away. Two questions replace it: which listings behave alike, and what few forces drive nine Treasury yields. Because nothing checks the result automatically, the day is mostly about designing the check.
The datasets
The same file is used on more than one day. Each has one page, giving the question it answers, every column and when it is known, where it came from, what was done to it, and what it does not establish.
| London listings | 54,636 rows, 11 columns. Used on Day 2a, Day 2b, Day 3, Day 4. |
| Bank marketing calls | 41,188 rows, 22 columns. Used on Day 2a, Day 2b, Day 3. |
| Recession panel | 521 rows, 6 columns. Used on Day 3. |
| US Treasury yield curve | 9,143 rows, 10 columns. Used on Day 4. |
| London listings, foundations extract | 20,740 rows, 17 columns. Used on Day 1. |
What the notebooks do
They teach the lecture rather than solve a project. Each idea is computed from its definition on a case small enough to check by hand, and only then applied at full scale: a line fitted on four properties before 41,188 bank contacts, a tree split scored by hand before a forest of 300, K-means run on five numbers before 32,749 listings.
Each notebook is saved with every output in place, so you can read a
whole session without running anything. They need Python with
numpy, pandas, matplotlib,
scipy and scikit-learn. Nothing is downloaded at
run time.