Day 1 · Monday · Empirical studio

Extracting only what a source states

Gate 1: answer before code

1. The claim this dataset can test

   Which claim matches the recorded target?

   1. The system discovers each speaker's actual authority inside the institution

   2. The supplied description contains evidence for one permitted role label, or the system abstains

   3. The speaker's job title causes the subject of the speech

   4. The author's name is enough to infer seniority

2. Permitted information

   Which candidate input belongs in this exercise?

   1. An outside biography found by searching the author's name

   2. A gender inferred from the author's name

   3. An exact role phrase in the supplied public description

   4. A private institutional personnel record

3. Why the audit labels stay closed

   Why must the rule be frozen before audit labels are opened?

   1. So that audit records can be treated as if they had no descriptions

   2. So that the audit evaluates a rule not selected to fit those same outcomes

   3. Because development labels are always more accurate than audit labels

   4. Because a frozen rule needs no human review

What you will do in the notebook

1. Load the 600 development records and confirm which columns are available.

2. Use the most common resolved development label as the full-coverage baseline.

3. Run the compact phrase rule. Inspect development mismatches and make no more than one change supported by development evidence.

4. Freeze both systems and generate predictions for all 250 audit IDs.

5. Confirm that the evidence span copied by the rule occurs in the supplied description.

6. After I release the aggregate audit table, answer Gate 2. Do not revise either system on the audit results.

1. Comparing the two systems

   Which statement reads all relevant denominators correctly?

   1. The compact rule is correct on every possible BIS description

   2. The plurality baseline has better performance because it answers more records

   3. Against the provisional audit reference, the compact rule answers 245 of 250 and is correct on those 245; five records are abstentions

   4. Selective accuracy and coverage are the same quantity

2. A defensible workflow

   How should an answered record and an abstention be handled?

   1. Treat every answer as final and delete abstentions

   2. Preserve the source phrase, allow review or correction, and route abstentions or disputes to a person

   3. Infer a missing role from the author's institution

   4. Replace abstentions with the plurality class so the compact rule reports 100% coverage

3. What independent review protects

   Suppose the automatic reference extractor and the compact rule share similar phrases. Why is blinded human review important?

   1. Agreement may reflect shared construction rather than independent evidence of correctness

   2. Human review will necessarily make coverage equal to 100%

   3. Blinding allows reviewers to tune the compact rule

   4. A reviewed label becomes a causal effect

4. The narrow conclusion

   Which conclusion is appropriate at the end of today's studio?

   1. The rule identifies speakers' true institutional power

   2. The build evidence shows that a frozen phrase rule can extract supported labels from this audit sample with reported coverage, pending independent label review

   3. The rule will retain the same performance on every institution and future period

   4. The deployment split may now be used to revise the rule

← Day 1 lecture  ·  All four days →