Why train a writer
Language models now help economists revise drafts, yet outside judges can still tell AI prose apart from what economists write almost every time. Even when a frontier model receives a detailed writing guide, a cliché list and three published examples, an outside judge still picks the authors' paragraph in 119 of 120 blind comparisons.
Generic editing models smooth away concrete habits of economics writing. They replace field terms with casual phrases, spell out transitions that economists leave implicit, and even out sentence rhythm. Many of those conventions are hard to state as rules, so this project tests whether an open model can absorb them directly from pre-2023 economics paragraphs and adversarial feedback.
How the two-stage system works
Everything trains on a single rented 80 GB GPU using one 4-bit Gemma 4 31B base model and three low-rank adapters: a writer, a frozen Step 1 anchor, and a co-trained internal judge. Total API spend across pilots, test drafts, outside judges and content checks was 131 US dollars.
Build a pre-LLM corpus
Collect 2,120 economics and finance papers dated 1995 to 2022, mostly open working papers before copy-editing, and screen 5,267 paired AI drafts and authors' paragraphs holding fixed facts.
Supervised training (Step 1)
Fine-tune the open 31B writer on the 5,267 pairs to absorb pre-2023 prose, and pre-train the internal judge on 1,225 separate clean pairs until it reaches 99.0 per cent held-out accuracy.
Adversarial updates (Step 2)
Sample four revisions per unseen draft at temperature 0.7, score each in both display orders against the internal judge, remove the within-group length slope, and update the writer with group-relative policy optimisation while replaying pairs to the judge.
Independent evaluation
Test all checkpoints blind in both display orders under GPT-6.1 Sol and Claude Opus 5.5 on 137 post-cut-off 2026 NBER introduction paragraphs and 54 recall-screened 1999 to 2022 paragraphs, plus blind content-fidelity audits and a human reading test.
To stop the writer from winning simply by writing longer paragraphs, Step 2 adjusts each candidate's clipped log-odds score by the within-group slope on relative log length before computing the group-relative reward:
\[ r_i = \operatorname{clip}\!\left(\log\frac{\bar{p}_i}{1-\bar{p}_i},\,-6,\,6\right) - \hat{\gamma}\,\ell_i, \qquad \ell_i = \log\frac{w_i}{w_{\text{draft}}}. \]
Main findings
On 137 introduction paragraphs from 2026 NBER working papers written after the base model's cut-off, raw GPT-6.1 Sol drafts are taken for the authors in 6.9 per cent (GPT judge) and 8.4 per cent (Opus judge) of comparisons. Prompted Claude Opus 5.5 with our writing guide reaches 21.2 per cent under GPT and 7.7 per cent under Opus. Supervised training lifts the open 31B model to 40.1 and 40.5 per cent, and adversarial training raises it to 70.8 and 55.8 per cent in Run 1 (68.2 and 54.4 per cent in Run 2).
Across all 191 test paragraphs and both judges, adversarial training adds 21.1 percentage points in Run 1 (95 per cent interval 15.4 to 26.6) and 20.8 points on a fresh seed in Run 2 (15.6 to 26.0). Both runs pass all five pre-registered decision rules. Mean length shifts by at most 1.4 per cent, and paragraphs within 5 per cent of draft length keep gains of 21.4 and 19.3 points. Both final writers preserve every number in all 137 paragraphs from 2026.
| Version | 2026 NBER (GPT) | 2026 NBER (Opus) | 1999–2022 (GPT) | 1999–2022 (Opus) |
|---|---|---|---|---|
| Raw AI draft (GPT-6.1 Sol) | 6.9% | 8.4% | 0.0% | 0.0% |
| Guided Claude Opus 5.5 | 21.2% | 7.7% | 1.9% | 0.9% |
| Supervised 31B (Step 0, Run 1) | 40.1% | 40.5% | 11.1% | 13.9% |
| Adversarial 31B (Run 1, final) | 70.8% | 55.8% | 31.5% | 25.9% |
| Adversarial 31B (Run 2, final) | 68.2% | 54.4% | 25.9% | 25.0% |
A small open human reading test points in the same direction. Across 23 readers and 95 blind answers on 24 paragraphs from 2026, readers picked the authors' paragraph 61.8 per cent of the time against raw drafts, compared with 37.9 per cent against the supervised writer and 43.8 per cent against the adversarial writer (pooled difference 20.8 percentage points, 95 per cent interval 6.9 to 35.1).
Content fidelity and the trade-off
Keeping numbers is not enough if a revision changes what a paper claims. Two blind annotators checked every version across all 191 paragraphs against its input draft for dropped or added claims, changed hedges, entity errors and meaning shifts. Both annotators find that trained writers alter claims more often than guided Opus (28.3 and 6.8 per cent supervised against 8.4 and 0.5 per cent guided).
In Run 1, the upper bound of GPT's paired increase over the supervised writer reaches 12.3 percentage points, so the pre-specified no-cost threshold fails. Even when every major flag is counted as a failure, however, Run 1's adversarial gain remains clearly positive: +14.1 percentage points under GPT flags and +19.1 points under Opus flags. In practice, trained writers get much closer to economists' prose than a style guide, but researchers still need to check meaning before keeping a revision.
Reproduce every table and figure
Third-party authors' paragraphs cannot be redistributed in full, so the public repository provides every anonymised pairwise verdict, content-fidelity rating, training log, word count, number-retention indicator and human reading-test answer alongside the complete training, evaluation and replication scripts. One command reproduces every table, confidence interval, decision rule and figure in the paper.
git clone https://github.com/kansoy/ai-economist.git
cd ai-economist
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python3 run_all.pyLimits
Several limits qualify the results. Main attribution scores rely on two frontier judge families, and GPT-6.1 Sol wrote the input drafts. The human reading test is small (23 readers, 95 answers) with unverified identities. The task revises single introduction paragraphs with fixed content rather than drafting a paper from notes. Some 2026 working papers may already carry light AI editing, while older pre-2023 paragraphs stay well below chance for every system. Finally, both adversarial runs stopped once the 31B internal judge hit its accuracy floor, so testing a larger judge is a natural next step.
The working paper is at fatih.ai/ai_economist.pdf, the full paper summary is on the research page, and the replication code and anonymised evaluation data are released under the MIT licence on GitHub.