Day 3 · Wednesday · Empirical studio
Can pre-departure information rank late arrivals?
Do not open the October–December outcomes until your model, information set, and primary measure are fixed. A high score cannot make after-departure information valid at a pre-departure clock.
Dataset card
Official BTS monthly files; monthly-balanced teaching sample
| Your files | bts_2025_development.csv;
bts_2025_audit_features.csv; day3_application.ipynb |
|---|---|
| Empirical question | Among flights that later completed, were not diverted, and have an observed outcome, can information available before scheduled departure rank arrival at least 15 minutes late? |
| One row | One reported domestic flight in the stated retrospective study population. |
| Period and sample | January–December 2025; 3,000 deterministically selected eligible flights per month. |
| Outcome | delayed_15: arrival at least 15 minutes late. |
| Prediction time | Before scheduled departure. |
| Development | January–September, using expanding earlier-to-later folds. |
| Final audit | October–December; outcome hidden until the system is frozen. |
| Source | U.S. Department of Transportation, Bureau of Transportation Statistics. |
| Used in | Day 3 concept studio Questions 8–10; empirical Questions 1–6; Day 3 problem set Questions 1–12. |
Variables
| Field | Meaning | Use |
|---|---|---|
row_id | Stable schedule identity | Alignment only |
Month, DayOfWeek | Scheduled calendar fields | Eligible |
Reporting_Airline | Reporting carrier | Eligible |
Origin, Dest | Scheduled airports | Eligible |
scheduled_departure_minutes | Planned departure time | Eligible |
scheduled_arrival_minutes | Planned arrival time | Eligible |
CRSElapsedTime | Planned elapsed minutes | Eligible |
Distance | Scheduled distance | Eligible |
DepDelayMinutes | Realised departure delay | Too late |
Cancelled, Diverted | Later status | Define exclusions; not predictors |
delayed_15 | Arrival at least 15 minutes late | Target; hidden in audit |
split | Development or audit | Partition only |
Before you run code
- Which claim matches these rows?
A live queue covering every scheduled flight.
A retrospective ranking among flights later known to be completed, non-diverted, and labelled.
The causal effect of contacting a flight.
Annual U.S. delay prevalence.
- Which feature must be excluded?
Planned distance.
Scheduled arrival time.
Reporting carrier.
Realised departure delay.
- Why use rolling months?
To guarantee 2026 performance.
To make the validation exercise resemble fitting on the past and using the model later.
To ensure every fold has the same prevalence.
To maximise training fit.
Notebook order
Inspect the monthly sample and the target prevalence.
Separate fields known before departure from fields observed later.
Fit preprocessing inside each training window.
Select tree depth using mean fold AP.
Compare the schedule prior, logistic model, selected tree, and forest on identical rolling predictions.
Save the selected model and primary measure.
Open October–December outcomes once and complete Questions 4–6.
After the audit outcomes are opened
- Interpreting the final AP
The frozen forest has audit AP 0.3325 and audit prevalence 0.2251. Which statement is best?
The ranking improves on a no-skill AP reference within this audit population.
The model prevents 33.25% of delays.
The model is 33.25% accurate.
The model is guaranteed to retain the same AP in 2026.
- Interpreting the top decile
Among the 900 highest-scored audit-study flights, 350 are late. Which calculation is the relevant precision?
\(350/900\).
\(350/9{,}000\).
\(2{,}026/900\).
\(900/2{,}026\).
- Can the top 900 become tomorrow's queue?
Yes; the ranking has already been audited.
Yes; every audit flight was observed by BTS.
No; membership in the completed, non-diverted, labelled population is known only after the flight, so a prospective study must include all cases eligible at the decision time.
No; AP can never be used for operations.
