Day 1 · Monday · Problem set

Prediction claims, baselines, and evaluation

1. Precision of the review queue

   What is the rule's precision?

   1. \(30/40=75\%\)

   2. \(30/50=60\%\)

   3. \(40/500=8\%\)

   4. \(50/500=10\%\)

2. Recall of late parcels

   What is the rule's recall?

   1. \(20/50=40\%\)

   2. \(30/40=75\%\)

   3. \(30/50=60\%\)

   4. \(450/500=90\%\)

3. A capacity-matched baseline

   Randomly selecting 40 of the 500 audit parcels would contain an expected \(40\times 50/500=4\) late parcels. What follows?

   1. The rule's queue prioritises late parcels much better than random selection on this audit

   2. The rule prevents exactly 26 late deliveries

   3. Accuracy must be 96% because 40 records were flagged

   4. The rule is ready for automatic rerouting without a cost or intervention study

1. Information at the stated prediction moment

   Which variable is invalid for the before-term score?

   1. Prior completed course results

   2. Programme and study mode recorded before term

   3. A support appointment completed in week three

   4. A declared support preference recorded before term

2. Target and decision are different

   The score predicts academic difficulty accurately. What does that fact alone not establish?

   1. Whether the difficulty outcome can be defined

   2. Whether a voluntary support invitation improves outcomes

   3. Whether one row represents one student

   4. Whether earlier course results were recorded

3. A responsible use

   Which proposed use stays closest to the stated design?

   1. Automatically remove high-scoring students from the programme

   2. Offer a reviewable, voluntary invitation and monitor who is missed or burdened

   3. Publish individual scores to classmates

   4. Treat the score as proof that a student will struggle

1. Evaluation matched to deployment

   Which test best represents the stated 2027 use?

   1. Random rows from old documents, with the same organisations in training and testing

   2. Training performance after model selection

   3. Later documents from organisations kept entirely out of development

   4. Only the easiest documents from the new organisations

2. A risk that remains

   Even with later documents from unseen organisations, what uncertainty remains?

   1. Whether conditions in 2027 differ from the evaluation period

   2. Whether the training rows have labels

   3. Whether the classifier has an output

   4. Whether unseen organisations are absent from development

3. Using an audit twice

   The team studies errors on the final evaluation, changes the rule, and reports the revised score on the same rows. Which statement is correct?

   1. The rows now informed development, so fresh untouched cases are needed for a final check

   2. The revised score remains fully independent because no new feature was added

   3. The audit becomes larger after the revision

   4. A higher revised score proves performance in 2027

1. Coverage and selective accuracy

   A system answers 95 of 100 records and is correct on 93 answered records. Which pair is correct?

   1. Coverage \(93\%\); selective accuracy \(95\%\)

   2. Coverage \(95\%\); selective accuracy \(93/95\)

   3. Coverage \(100\%\); selective accuracy \(93\%\)

   4. Coverage \(95/93\); selective accuracy \(95\%\)

2. A silent description

   The description gives no explicit role phrase. What output follows from the task definition?

   1. Infer the most likely role from the person's name

   2. Search an outside biography

   3. Return ABSTAIN and route the record for review if an answer is required

   4. Use the institution's most common role

3. Do label frequencies describe the institution?

   The extractor assigns SENIOR_STAFF more often in descriptions from one institution. Which conclusion is supported?

   1. The institution promotes more staff into senior roles

   2. The observed descriptions contain the extracted phrase more often; promotion practices were not measured

   3. Senior staff cause more speeches to be published

   4. The classifier has measured institutional fairness

← Day 1 lecture  ·  All four days →