Day 2 · Tuesday · Concept studio

When a prediction becomes an action

How this block runs

Choose each answer independently. I shall collect the first vote, give you two minutes to compare reasoning with a neighbour, and collect a second vote. Open Sheets B and C only when instructed.

1. Which loss reflects the stated concern?

   A dispatch centre says that one exceptionally large prediction error can close operations, even if ordinary errors are small. Which comparison most directly reflects that concern?

   1. RMSE, because squaring gives unusually large errors greater influence

   2. MAE, because absolute error always ignores small errors

   3. Accuracy, because the outcome is monetary

   4. The training mean, without any held-out evaluation

2. An association does not prescribe a change

   A listing model has a positive coefficient on accommodated guests. Which claim needs evidence beyond this predictive fit?

   1. Larger recorded capacity is associated with a higher fitted value, conditional on represented inputs

   2. The fitted equation can produce a prediction for a represented listing

   3. Increasing the same listing's capacity will cause its achievable price to rise

   4. The coefficient is expressed in pounds per recorded guest

3. Ranking and calibrated probability serve different uses

   Model A ranks fraud cases slightly better, but scores near 0.08 are fraudulent about 0.16 of the time. Model B ranks slightly worse, but its held-out probabilities match observed rates. Which pairing is most defensible?

   1. A for a fixed-capacity queue; B for comparing expected monetary costs

   2. B for every use because calibration is the only model property

   3. A for every use because ranking is the only model property

   4. Neither model can ever support a decision

4. The larger probability is not automatically the clearer action

   A payment has fraud probability 0.08. Customer R has response probability 0.30 if contacted. Which decision can be made from those probabilities alone?

   1. Review the payment because 0.08 is positive

   2. Contact R because 0.30 is larger

   3. Make both decisions using a 0.50 threshold

   4. Neither; the fraud case needs error costs and the outreach case needs no-contact outcomes and values

1. The cost-based threshold

   Equating the two costs gives \(\tau=3/(3+120)\approx0.0244\). What should happen at \(p=0.08\)?

   1. Pass, because 0.08 is below 0.50

   2. Review, because 0.08 exceeds the cost-based threshold

   3. Review only if the probability equals 1

   4. No decision can use monetary costs

2. A local calibration check

   Among 100 held-out payments scored near 0.08, 14 are fraudulent. What is the best interpretation?

   1. The model appears to underpredict risk in this score region; other regions and sampling uncertainty remain to be checked

   2. The entire model is proven perfectly calibrated

   3. The model overpredicts because 0.14 is larger than 0.08

   4. The review threshold must be changed to 0.50

1. Who should be contacted?

   Which action follows from the table?

   1. Contact R only because 0.30 is the largest response probability

   2. Contact S only because the estimated incremental value is positive

   3. Contact both because both contact probabilities exceed zero

   4. Contact neither because neither response probability exceeds 0.50

2. Evidence for uplift

   Which study most credibly estimates both contact and no-contact outcomes?

   1. Observe only customers selected by staff for contact

   2. Randomly assign eligible customers to contact and no contact, then evaluate on held-out assignments

   3. Fit a more flexible response model only among contacted customers

   4. Contact the customers with the largest predicted response and treat response as uplift

1. Two probabilities, two decisions

   Which statement is correct?

   1. Review the 8% fraud case because its risk exceeds the cost-based threshold; do not contact R because its estimated uplift does not cover contact cost

   2. Pass the fraud case and contact R because only probabilities above 0.50 matter

   3. Review the fraud case and contact R because both probabilities are positive

   4. Use the larger of 0.08 and 0.30 for both decisions

← Day 2 lecture  ·  All four days →