Fatih Kansoy

Projects · Paper, code and data · 2026

AI Economist

A working paper, single-GPU training pipeline and open replication package asking whether an open language model can learn economists' prose directly from pre-2023 papers and adversarial feedback without explicit style rules.

Status
Working paper, November 2026
Base model
4-bit Gemma 4 31B (single 80 GB GPU)
Corpus
2,120 pre-2023 papers, 5,267 training pairs
Evaluation
191 test paragraphs, 2 outside judges, 23 readers
Licence
MIT (code and replication data)
One base-model slab carrying writer, anchor and judge adapters, with a revised sheet rising toward the judge
Contents
  1. Why train a writer
  2. How the two-stage system works
  3. Main findings
  4. Content fidelity and the trade-off
  5. Reproduce every table and figure
  6. Limits

Why train a writer

Language models now help economists revise drafts, yet outside judges can still tell AI prose apart from what economists write almost every time. Even when a frontier model receives a detailed writing guide, a cliché list and three published examples, an outside judge still picks the authors' paragraph in 119 of 120 blind comparisons.

Generic editing models smooth away concrete habits of economics writing. They replace field terms with casual phrases, spell out transitions that economists leave implicit, and even out sentence rhythm. Many of those conventions are hard to state as rules, so this project tests whether an open model can absorb them directly from pre-2023 economics paragraphs and adversarial feedback.

How the two-stage system works

Everything trains on a single rented 80 GB GPU using one 4-bit Gemma 4 31B base model and three low-rank adapters: a writer, a frozen Step 1 anchor, and a co-trained internal judge. Total API spend across pilots, test drafts, outside judges and content checks was 131 US dollars.

  1. Build a pre-LLM corpus

    Collect 2,120 economics and finance papers dated 1995 to 2022, mostly open working papers before copy-editing, and screen 5,267 paired AI drafts and authors' paragraphs holding fixed facts.

  2. Supervised training (Step 1)

    Fine-tune the open 31B writer on the 5,267 pairs to absorb pre-2023 prose, and pre-train the internal judge on 1,225 separate clean pairs until it reaches 99.0 per cent held-out accuracy.

  3. Adversarial updates (Step 2)

    Sample four revisions per unseen draft at temperature 0.7, score each in both display orders against the internal judge, remove the within-group length slope, and update the writer with group-relative policy optimisation while replaying pairs to the judge.

  4. Independent evaluation

    Test all checkpoints blind in both display orders under GPT-6.1 Sol and Claude Opus 5.5 on 137 post-cut-off 2026 NBER introduction paragraphs and 54 recall-screened 1999 to 2022 paragraphs, plus blind content-fidelity audits and a human reading test.

To stop the writer from winning simply by writing longer paragraphs, Step 2 adjusts each candidate's clipped log-odds score by the within-group slope on relative log length before computing the group-relative reward:

\[ r_i = \operatorname{clip}\!\left(\log\frac{\bar{p}_i}{1-\bar{p}_i},\,-6,\,6\right) - \hat{\gamma}\,\ell_i, \qquad \ell_i = \log\frac{w_i}{w_{\text{draft}}}. \]

Main findings

On 137 introduction paragraphs from 2026 NBER working papers written after the base model's cut-off, raw GPT-6.1 Sol drafts are taken for the authors in 6.9 per cent (GPT judge) and 8.4 per cent (Opus judge) of comparisons. Prompted Claude Opus 5.5 with our writing guide reaches 21.2 per cent under GPT and 7.7 per cent under Opus. Supervised training lifts the open 31B model to 40.1 and 40.5 per cent, and adversarial training raises it to 70.8 and 55.8 per cent in Run 1 (68.2 and 54.4 per cent in Run 2).

Blind authorship attribution across raw AI draft, guided Opus, supervised 31B and adversarial 31B on 2026 NBER working papers and 1999 to 2022 papers
Share of order-balanced blind comparisons where outside frontier judges take each AI version for the authors' text (95 per cent paper-clustered bootstrap intervals; random chance is 50 per cent).

Across all 191 test paragraphs and both judges, adversarial training adds 21.1 percentage points in Run 1 (95 per cent interval 15.4 to 26.6) and 20.8 points on a fresh seed in Run 2 (15.6 to 26.0). Both runs pass all five pre-registered decision rules. Mean length shifts by at most 1.4 per cent, and paragraphs within 5 per cent of draft length keep gains of 21.4 and 19.3 points. Both final writers preserve every number in all 137 paragraphs from 2026.

Version2026 NBER (GPT)2026 NBER (Opus)1999–2022 (GPT)1999–2022 (Opus)
Raw AI draft (GPT-6.1 Sol)6.9%8.4%0.0%0.0%
Guided Claude Opus 5.521.2%7.7%1.9%0.9%
Supervised 31B (Step 0, Run 1)40.1%40.5%11.1%13.9%
Adversarial 31B (Run 1, final)70.8%55.8%31.5%25.9%
Adversarial 31B (Run 2, final)68.2%54.4%25.9%25.0%

A small open human reading test points in the same direction. Across 23 readers and 95 blind answers on 24 paragraphs from 2026, readers picked the authors' paragraph 61.8 per cent of the time against raw drafts, compared with 37.9 per cent against the supervised writer and 43.8 per cent against the adversarial writer (pooled difference 20.8 percentage points, 95 per cent interval 6.9 to 35.1).

Human reading test showing share of answers picking the authors' paragraph against raw draft, supervised writer and adversarial writer
Blind human reading test across 23 readers and 95 answers (95 per cent reader-clustered bootstrap intervals; one-sided Fisher p = 0.042).

Content fidelity and the trade-off

Keeping numbers is not enough if a revision changes what a paper claims. Two blind annotators checked every version across all 191 paragraphs against its input draft for dropped or added claims, changed hedges, entity errors and meaning shifts. Both annotators find that trained writers alter claims more often than guided Opus (28.3 and 6.8 per cent supervised against 8.4 and 0.5 per cent guided).

In Run 1, the upper bound of GPT's paired increase over the supervised writer reaches 12.3 percentage points, so the pre-specified no-cost threshold fails. Even when every major flag is counted as a failure, however, Run 1's adversarial gain remains clearly positive: +14.1 percentage points under GPT flags and +19.1 points under Opus flags. In practice, trained writers get much closer to economists' prose than a style guide, but researchers still need to check meaning before keeping a revision.

Content-fidelity audit showing major content change rates and fidelity-adjusted attribution across systems
Left: share of the 191 test paragraphs flagged for a major content change by blind annotators. Right: attribution rate across both outside judges when every major flag counts as a failure.

Reproduce every table and figure

Third-party authors' paragraphs cannot be redistributed in full, so the public repository provides every anonymised pairwise verdict, content-fidelity rating, training log, word count, number-retention indicator and human reading-test answer alongside the complete training, evaluation and replication scripts. One command reproduces every table, confidence interval, decision rule and figure in the paper.

Terminal · clone and reproduce all tables and figures
git clone https://github.com/kansoy/ai-economist.git
cd ai-economist
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python3 run_all.py

Limits

Several limits qualify the results. Main attribution scores rely on two frontier judge families, and GPT-6.1 Sol wrote the input drafts. The human reading test is small (23 readers, 95 answers) with unverified identities. The task revises single introduction paragraphs with fixed content rather than drafting a paper from notes. Some 2026 working papers may already carry light AI editing, while older pre-2023 paragraphs stay well below chance for every system. Finally, both adversarial runs stopped once the 31B internal judge hit its accuracy floor, so testing a larger judge is a natural next step.

The working paper is at fatih.ai/ai_economist.pdf, the full paper summary is on the research page, and the replication code and anonymised evaluation data are released under the MIT licence on GitHub.

More projects

All projects