Day 3 · Wednesday · MCQ problem set

Flexibility, ranking, and honest evaluation

After class12 questions; 24 marks

Remember

Choose the single best answer. The distractors are plausible mistakes, so check the prediction time, population, measure, and audit role before choosing.

Dataset

BTS 2025 domestic flights

Used in Questions 1–12

One row is one flight in the later-completed, non-diverted, observed-outcome study population. The target is arrival at least 15 minutes late. The prediction time is before scheduled departure. January–September is used for development and rolling validation; October–December is the untouched audit. The sample contains 3,000 eligible rows per month and is not weighted to represent annual flight volume.

Two splits improve training fit equally2 marks
One split creates a 23-flight leaf; another leaves every child above 100 flights. What should decide between them?
1. The split with the smaller leaf, because it is more specific.
2. The split that appears first in the software output.
3. Deployment-matched later performance and stability; if those are indistinguishable, prefer the simpler, better-supported partition.
4. Training fit alone, because both splits use the same outcome.
Fifty models on three folds2 marks
Why may the best of 50 validation scores be optimistic?
1. The winner can exploit favourable noise in folds repeatedly used for selection.
2. Validation scores are always lower than future scores.
3. Only the first model is permitted to use cross-validation.
4. Optimism occurs only when the target is continuous.
A carrier history feature2 marks
For an August flight, a carrier delay rate is calculated using outcomes through 31 August. What is wrong?
1. Carrier rates may never be predictors.
2. August outcomes enter predictions for August flights before those outcomes existed.
3. The feature is categorical rather than numeric.
4. The rate should use only delayed flights.
Preprocessing inside a time fold2 marks
What is the correct treatment of imputation and category definitions?
1. Learn them from the full year so every month uses identical values.
2. Learn them from training months inside each fold, then apply them to the later judge months.
3. Learn them from the audit period only.
4. Skip validation whenever a category is missing.
Complexity under a tie2 marks
Two models have indistinguishable later AP. Which is the best default?
1. The more complex model because it has greater capacity.
2. The simpler model, unless the flexible model has a predeclared operational benefit supported by fresh evidence.
3. Whichever has higher training AP.
4. Average their audit outcomes before choosing.
AP and the highest decile disagree2 marks
The forest has higher AP; logistic regression has higher top-decile precision. Which statement is correct?
1. The results are impossible because rankings cannot cross.
2. AP values ordering across the ranking, while top-decile precision values one cutoff; the declared use determines the relevant winner.
3. Logistic regression must win because it is simpler.
4. Forest must win every decision because it has higher AP.
A high score with late information2 marks
An after-departure model has AP 0.8891. How should it be reported?
1. As the winning pre-departure model.
2. As evidence that leakage is harmless.
3. As a result for a different, later information set; it cannot enter the pre-departure comparison.
4. As proof that departure delay causes arrival delay.
Moving the decision time2 marks
A manager moves the score to after pushback so realised departure delay may be used. What follows?
1. Nothing changes except model accuracy.
2. This is a new decision problem with a later action, population, baseline, and evaluation.
3. The original audit automatically validates the new use.
4. The result now estimates the effect of rerouting.
Monthly-balanced sampling2 marks
What may be concluded from exactly 3,000 eligible flights per month?
1. Their pooled delay share is the annual U.S. delay prevalence.
2. Every month has equal national flight volume.
3. Model comparisons are possible within this teaching sample, but annual prevalence needs appropriate population sampling or weights.
4. Seasonal change has been removed.
An ex-post population2 marks
Why is the audit top decile not yet a live queue?
1. A queue can never be based on predictions.
2. Completion, diversion, and label availability are known only later, so the desk cannot identify the same eligible population before departure.
3. The audit has fewer than 10,000 flights.
4. Top-decile precision is not a percentage.
Revising after the audit2 marks
After seeing audit errors, the team changes a feature and reports a new score on the same audit. What is now required?
1. Nothing; the higher score validates the change.
2. A fresh untouched audit, because the old audit has become development information.
3. Only a new training score.
4. Removal of every feature used by the first model.
A cancellation study2 marks
Which redesign is coherent?
1. Keep only completed flights and reuse arrival delay as cancellation.
2. Use one scheduled flight as the unit, include it before cancellation is known, define cancellation by a stated horizon, score before the action, and leave a label-arrival gap before the final audit.
3. Label every cancelled flight as on time.
4. Predict cancellation after the cancellation code is recorded.

← Day 3 lecture · All four days →

Oxford · United Kingdom Course home Teaching CV