Benchmark · 10 documents · gold v3

PDF extraction

Schema-guided extraction on an attributed ExtractBench subset: ten documents, gold adjudicated in the open.

Measures Value F1 per document: every gold cell matched or not after date normalization, repeated records aligned by content, extra rows penalized. 0 to 1, no partial credit per cell.

Cases 10 documents finance 2 · energy 1 · government 2 · real estate 1 · automotive 1 · healthcare 1 · supply chain 1 · legal 1 7 short · 2 medium · 1 long · 8 real, 2 synthetic

Gold v3 · adjudicated 4 disputes accepted, 9 cells changed · 2026-09-07

Systems 4 on the board Coding agents as extractors · restricted document reader

How sure Top system ranges 0.964 to 1.000 across documents. With 10 documents, one document moves a mean by up to 0.10. Read the matrix, not just the mean.

Last change 2026-09-07

Results · value f1 on gold v3

Mean over the documents that have gold. A timeout or error counts as 0; a document without gold is coverage, never a score. Light dot: gold v1. Dark dot: gold v3 after every accepted dispute.

gold v1 gold v3

System	Mean · v3	· v1	Graded	Complete	Timeouts	Median time	Cost
Codex · GPT-5.6 Sol · medium effort codex-cli 0.153.4 · code execution on	0.989	0.958	10 / 10	9 / 10	0	59s	unknown
Codex · GPT-6 Astra · medium effort codex-cli 0.153.4 · code execution on	0.894	0.863	9 / 10	9 / 9	1	43s	unknown
Claude Code · Claude Opus 5 · medium effort 2.1.263 (Claude Code) · reader only	0.887	0.871	10 / 10	9 / 10	0	30s	unknown
Claude Code · Claude Sonnet 5 · medium effort 2.1.263 (Claude Code) · reader only	0.881	0.852	9 / 10	9 / 9	1	38s	unknown

What was fixed in the gold

4 disputes accepted, 9 gold cells changed. Earlier labels stay as earlier versions; every saved output was regraded against each. Nothing was rerun.

Dispute 001 · accepted · CDW Government invoice

Invoice vendor phone: account manager's direct line versus null

	gpt-6-astra	gpt-5.6-sol	claude-opus-5	claude-sonnet-5
before	0.977	0.955	0.955	0.886
after	1.000	0.977	0.977	0.886

Dispute 002 · accepted · Byline Bancorp earnings report

Byline Bancorp Q2 2023: headline rounding rule and pre-tax income labeled as operating income

	gpt-6-astra	gpt-5.6-sol	claude-opus-5	claude-sonnet-5
before	0.714	0.714	0.786	0.714
after	1.000	1.000	0.929	1.000

Dispute 003 · accepted · Unclaimed property list · synthetic

Unclaimed property list: owner names wrapped onto two lines were truncated to the second line

	gpt-6-astra	gpt-5.6-sol	claude-opus-5	claude-sonnet-5
before	0.000	0.979	0.032	0.000
after	0.000	0.979	0.032	0.000

Dispute 004 · accepted · Customs entry summary

Customs entry summary: the relationship indicator printed at the start of the second and third invoices

	gpt-6-astra	gpt-5.6-sol	claude-opus-5	claude-sonnet-5
before	0.995	0.995	0.995	0.995
after	0.997	0.997	0.997	0.997

Every document, every system

Click a cell to see the field-by-field comparison against the page. Striped cells are timeouts; the ⚠ marks a submission with fewer rows than gold.

Document	gpt-6-astra	gpt-5.6-sol	claude-opus-5	claude-sonnet-5
CDW Government invoice finance · short	1.000	0.977	0.977	0.886
Pepco residential bill energy · short	1.000	1.000	1.000	1.000
Customs entry summary government · short	0.997	0.997	0.997	0.997
Mortgage closing disclosure real estate · short	0.984	0.997	0.984	0.962
Chevrolet vehicle valuation automotive · short	1.000	0.991	0.991	1.000
Medicaid remittance advice healthcare · short	0.964	0.964	0.964	0.964
Caterpillar excavator specifications supply chain · short	0.992	1.000	1.000	1.000
Byline Bancorp earnings report finance · medium	1.000	1.000	0.929	1.000
Service list · synthetic legal · medium	1.000	0.984	1.000	0.999
Unclaimed property list · synthetic government · long	timeout	0.979 ⚠	0.032 ⚠	timeout

Changes

2026-09-07 gold v3 · adjudicated · v2 plus disputes 003 and 004: two wrapped owner names, two relationship indicators.
2026-09-07 gold v2 · adjudicated · v1 plus disputes 001 and 002, adjudicated from page evidence.
2026-09-07 gold v1 · imported · Upstream ExtractBench annotations at the pinned revision; not independently reviewed.
2026-09-06 run · Codex · GPT-6 Astra · medium effort, Codex · GPT-5.6 Sol · medium effort, Claude Code · Claude Opus 5 · medium effort, Claude Code · Claude Sonnet 5 · medium effort · 40 attempts

Run it on your documents

Fork this benchmark, add your PDFs, run any system, confirm gold from the page. Nothing leaves your machine.

pip install benchmax
benchmax init my-pdf-extraction --from pdf-extraction
cd my-pdf-extraction && benchmax add ./my-pdfs
benchmax run --target extract.py:extract
benchmax serve

Fix this benchmark

Think a label is wrong? Open the document, click the field, and file a dispute with the page evidence. Accepted disputes become a new gold version and every saved output is regraded, never rerun.

Dispute a label Add a document or a system