Day 3 · Wednesday · In-class concept studio

Make flexibility earn its place

Setting. A streaming service scores each free-trial user at the end of the first 48 hours. The target is whether the user pays when the seven-day trial ends. The service can contact only the highest-scored fifth.

1. Prediction is not programme impact

   Which claim can this study evaluate without an experiment?

   1. Which users are most likely to pay at the end of the trial.

   2. Which users will be caused to pay by receiving outreach.

   3. How much revenue the outreach programme creates.

   4. Whether contacting everyone is socially desirable.

2. A feature must exist at the cutoff

   Which proposed input violates the 48-hour prediction clock?

   1. Sign-up device.

   2. Minutes streamed during the first 48 hours.

   3. Referral channel.

   4. Total minutes streamed during all seven trial days.

3. Choose depth from later folds

   The same time-ordered folds judge every depth.

   Depth	Fold 1 AP	Fold 2 AP	Fold 3 AP	Mean AP
   2	0.24	0.28	0.26	0.260
   4	0.30	0.31	0.29	0.300
   6	0.34	0.30	0.31	0.317
   8	0.36	0.26	0.25	0.290

   Which depth should advance under the declared mean-AP rule?

   1. Depth 2, because it is simplest regardless of performance.

   2. Depth 4, because all fold scores are below 0.32.

   3. Depth 6, because it has the highest mean and does not rely on one fold.

   4. Depth 8, because it has the largest single-fold score.

4. The measure follows the use

   System	Pooled AP	Precision in highest-scored 20%
   Logistic	0.31	0.41
   Tree	0.34	0.37
   Forest	0.35	0.38

   Which statement is correct?

   1. Forest wins a predeclared AP comparison; logistic wins a predeclared highest-fifth comparison.

   2. Forest is best for every possible decision.

   3. Logistic must be rejected because its AP is below the forest's.

   4. The table proves outreach causes payment.

Setting. Ten manufactured units completed a 30-day observation period; two failed. Systems A and B rank units from highest to lowest risk. The desk can inspect three units.

1. Choose for the three-unit queue

   Which system serves this inspection capacity better?

   1. System A, because its first failure is ranked first.

   2. System B, because both failures are inside the three-unit queue.

   3. They are identical because both rankings contain the same ten units.

   4. Neither can be compared without training accuracy.

2. A global measure need not settle a cutoff

   Suppose the two systems have equal AUC. What does that establish?

   1. They must identify the same three units.

   2. They have the same rate of correct failure–non-failure pair ordering, but may differ at the operating cutoff.

   3. They have equal calibration.

   4. They will cause the same number of failures to be prevented.

3. Lift is not an intervention effect

   System B finds both failures among three inspections. Overall prevalence is \(2/10\). Which interpretation is valid?

   1. Top-three precision is \(2/3\), and lift over prevalence is \((2/3)/(2/10)\); neither number is the effect of inspection.

   2. Inspection prevents \(2/3\) of all failures.

   3. AUC equals \(2/3\).

   4. The model is calibrated because two failures appear near the top.

1. Can the 0.71 model enter the comparison?

   1. Yes; a higher AP overrules the prediction clock.

   2. Yes; leakage matters only for causal studies.

   3. No; it answers a later-information question and is ineligible for the 48-hour decision.

   4. No; forests are always too complex.

2. What can the high score still tell us?

   1. Nothing; the number must be false.

   2. It may be a valid score for a new decision made after the seven-day trial, provided that decision and evaluation are redefined.

   3. It validates the original outreach queue.

   4. It shows that total viewing causes payment.

3. The final audit

   After selecting features, depth, and metric on rolling folds, what is the proper use of October–December outcomes?

   1. Open them repeatedly until the model improves.

   2. Use them once to judge the frozen system, then collect fresh data for any revision inspired by the result.

   3. Add them to every validation fold before choosing depth.

   4. Use only the best-performing month.

← Day 3 lecture  ·  All four days →