Four-day course AI & Machine Learning

Day 4 · Thursday · Empirical studio

Can a retrieved passage support the answer?

Read before opening the notebookBIS speech retrieval corpus
Important

Keep the reference judgements closed until both complete rankings are saved. A similarity score determines rank; it does not determine whether a passage supports a sentence.

Dataset card

Dataset
BIS speech retrieval corpus

Pinned BIS speech collection; no live web search

Your filesbis_speeches.csv, bis_chunks.csv, queries.csv, day4_application.ipynb
Empirical questionCan lexical and latent-semantic retrieval place a course-authored reference passage near the top, and can the retrieved words support a supplied candidate sentence?
One ranked itemOne 300-whitespace-token speech chunk with 50-token overlap and at least 40 tokens in a final chunk.
Corpus40 speeches dated 2018–2025, producing 474 chunks.
QueriesTwelve fixed questions labelled Q01–Q12.
MethodsBM25 over exact terms; TF–IDF plus truncated SVD and cosine similarity as an inspectable LSA-style representation.
Reference evidenceKnown-item judgements released only after the rankings are frozen; not exhaustive relevance labels.
CitationBIS:<speech_id>:<four-digit chunk number>.
SourceBank for International Settlements central-bank speeches, pinned through 20 June 2025.
Used inDay 4 concept studio Questions 5–10; empirical Questions 1–6; Day 4 problem set Questions 5–10.

Files and fields

File or fieldMeaningUse
bis_speeches.csvSpeech-level title, author, date, URLProvenance and inspection
bis_chunks.csvCitation ID, speech ID, chunk number, title, textCandidate collection
queries.csvQ01–Q12 and fixed question textInput to both methods
student_frozen_rankings.csvQuery, method, rank, citation, scoreSaved before qrels
Released qrelsKnown reference chunks for supported questionsPost-freeze scoring

Before you run code

  1. What can known-item retrieval evaluate?

    1. Whether a named reference chunk appears near the top of this fixed collection.

    2. Whether every answer is true on the open web.

    3. Whether acting on an answer is fair.

    4. Whether the generated sentence is causally correct.

  2. Why must both methods rank the same chunks?

    1. So differences can be attributed to representation and scoring rather than unequal candidate collections.

    2. So both methods must produce identical rankings.

    3. So no relevance judgement is needed.

    4. So BM25 becomes a semantic method.

  3. Why freeze rankings before qrels?

    1. To stop you reading chunk text.

    2. To prevent the known answers from guiding rank adjustment and to identify the exact rankings being evaluated.

    3. To guarantee exhaustive relevance.

    4. To increase cosine similarity.

Notebook order

  1. Confirm that both methods receive the same 474 citation IDs.

  2. Compare the top five passages for Q01 and read the text, not only the score or title.

  3. Produce all \(12\times2\times474=11{,}376\) ranking rows.

  4. Save the rankings and record their digest.

  5. Open the reviewed known-item judgements.

  6. Report Hit@1, Hit@3, Hit@5, and MRR for supported questions.

  7. For one supplied candidate sentence, select either a top-four passage that supports every clause or NO EVIDENCE IN THE PROVIDED CORPUS.

After the reference judgements are opened

  1. Interpreting two retrieval summaries

    BM25 has Hit@5 0.90 and MRR 0.652; the latent method has Hit@5 0.80 and MRR 0.656. Which statement is correct?

    1. BM25 finds a known item within five ranks more often; the latent method places the first known item slightly earlier on average.

    2. The latent method retrieves more known items at every cutoff.

    3. BM25 produces more faithful answers.

    4. Both methods have exhaustive recall above 0.80.

  2. A formatted citation can still fail

    A sentence cites a top-four chunk, but one clause is absent from the cited words. What should happen?

    1. Keep the clause because the citation format is valid.

    2. Remove or qualify the unsupported clause; provenance is not entailment.

    3. Replace the citation with the highest score.

    4. Treat the clause as supported by the model's training data.

  3. A ranking without evidence

    Q11 returns four high-scoring chunks, but none contains the requested fact. What is the correct conclusion?

    1. The best-ranked chunk is automatically sufficient.

    2. Average the four chunks into an answer.

    3. Refuse within the supplied-corpus boundary; a ranker always orders candidates even when no candidate answers the question.

    4. Declare the question false in the outside world.

← Day 4 lecture  ·  All four days →

Oxford · United Kingdom Course home Teaching CV
University of Oxford