Day 4 · Thursday · MCQ problem set

Representation, evidence, and accountable use

After class12 questions; 24 marks

Remember

Choose the single best answer. Ask what has actually been measured: distance, variance, retrieval of a known item, support for a sentence, or consequences of a decision.

Dataset A: selected company returns

Dataset

Thirty-two U.S.-listed companies, 2016–2024

Used in Questions 1–4

For clustering, one case is one selected company represented by its standardised daily return history. For PCA, days are cases and companies are variables. Candidate clusterings have silhouettes 0.103–0.131; a six-cluster solution has sector-label agreement 0.553 and early/late stability 0.749. Stock PC1 explains 42% of variance with same-signed loadings. This convenience sample has survivorship bias.

Without standardisation2 marks
What is the main risk when company return histories are clustered without scaling?
1. High-volatility companies can dominate Euclidean distance.
2. Every company receives its own cluster.
3. Correlations become exactly zero.
4. PCA becomes supervised learning.
The same table, transposed2 marks
Why do company clustering and days-by-companies PCA answer different questions?
1. K-means and PCA cannot use the same numbers.
2. The case and variable roles change: one groups companies; the other describes directions of joint daily variation.
3. Transposition changes every return value.
4. PCA supplies true sector labels.
Three checks2 marks
Which interpretation uses all three reported checks correctly?
1. Silhouette proves the true number of sectors; the other checks are unnecessary.
2. Weak separation, moderate recognisability, and one encouraging time-stability check support a working descriptive partition, not a natural taxonomy.
3. Stability proves causal sector membership.
4. Agreement 0.553 means 55.3% prediction accuracy.
PC1 and beta2 marks
A colleague calls each PC1 loading the company's CAPM beta. What is wrong?
1. PCA loadings describe a variance direction in this sample; CAPM beta comes from a specified market-return relationship.
2. CAPM beta and PCA loadings are identical after standardisation.
3. A loading is a cluster label.
4. PC1 can explain only categorical outcomes.

Dataset B: BIS speech retrieval corpus

Dataset

Forty BIS speeches and 474 chunks

Used in Questions 5–10

BM25 and an LSA-style method rank the same 300-token chunks for twelve fixed queries. Released qrels identify course-authored known items but are not exhaustive relevance labels. A top-four citation is checked for provenance; you must still verify that the cited words support the answer.

Two correct implementations disagree2 marks
Which explanation is consistent with both rankers working correctly?
1. Exact terms favour BM25 while latent co-occurrence favours another chunk.
2. Identical candidates require identical scores.
3. One method must have read the qrels.
4. The corpus must contain duplicate citation IDs.
Representation or corpus coverage?2 marks
How can you distinguish the two failures?
1. If a supporting passage exists but ranks poorly, suspect representation; if it is absent from the corpus, reranking cannot recover it.
2. Compare only the highest numerical scores.
3. A semantic method guarantees coverage.
4. Ask whether the citation ID has four digits.
Known-item measures2 marks
Why must Hit@5 not be called exhaustive Recall@5 here?
1. Hit@5 is always identical to MRR.
2. The qrels identify known reference passages, not every passage a reader might judge relevant.
3. Recall can be computed only for classifiers.
4. Five is too small a cutoff.
Wrong retrieval, faithful summary2 marks
An irrelevant passage is retrieved and summarised without adding anything. Where is the principal failure?
1. Answer faithfulness only.
2. Retrieval or corpus coverage before generation.
3. Citation typography.
4. Threshold governance.
Correct retrieval, unsupported generation2 marks
A supporting passage is retrieved, but the answer adds a claim not present in it. What is the principal remedy?
1. Increase the retrieval score.
2. Add a second unrelated citation.
3. Compare each clause with the cited words and remove or qualify the unsupported claim.
4. Treat the model's fluency as evidence.
Why refusal remains necessary2 marks
Why can a no-answer query still receive high scores?
1. Similarity ranking orders the available candidates even when none supports the requested fact.
2. A high score proves the fact is false.
3. The qrels force every query to have an answer.
4. Refusal is permitted only when every score is zero.

Decision case: payment suspension

Dataset

Selective investigation outcomes

Used in Questions 11–12

A flag opens an investigation and temporarily suspends payment. Recall is equal across two groups, but false-positive rates differ sharply. Outcomes are observed mainly for flagged cases, and successful appeals are rising.

The next evidence repair2 marks
Which action addresses selective labels?
1. Record every unflagged outcome as negative.
2. Audit a defensible sample of unflagged cases and preserve unobserved outcomes as unknown.
3. Stop reporting group-wise rates.
4. Increase the threshold until selection rates match.
The defensible governance response2 marks
Which response is best supported by the stated risks?
1. Continue automatic suspension because recall is equal.
2. Use citation faithfulness as the threshold rule.
3. Pause automatic suspension or restrict the system to assisted referral; name a threshold owner, preserve appeal, repair labels, monitor group-wise harm, and define a pause trigger.
4. Remove group labels so unequal error rates cannot be measured.

← Day 4 lecture · All four days →

Oxford · United Kingdom Course home Teaching CV