PDF extraction › System
Claude Code · Claude Opus 5 · medium effort
Mean value f1 0.887 on gold v3 (0.871 on v1) · 10 of 10 graded · 0 timeouts · median 30s · cost unknown
System
- Model
- claude-opus-5
- Harness
- 2.1.263 (Claude Code)
- Effort
- medium
- Tools
- document_info, read_page_text, read_page_image, submit_extraction
- Code execution
- no, reader-only
- Timeout
- 600 s per document
- Attempts
- one per document, no retries
- Config hash
- 87db0da77d0a0ee8
Track
Coding agents as extractors · restricted document reader
- Each agent received only the unchanged schema and four document tools: document info, page text, page image, submit extraction. No shell, browser, filesystem or web access.
- Codex retained its native code-mode computation and could assemble large submissions programmatically; Claude Code ran reader-only. This is a configured-agent comparison, not an equal-tool model ranking.
- One attempt per document and configuration. Timeouts at 600 seconds score zero in the fixed-denominator mean and are shown as timeouts, never hidden.
- Cost is unknown: runs used existing subscription quotas. CLI dollar estimates are not invoices.
- Source schemas contain answer hints for some fields (see the schema audit). Gold v1 is the upstream annotation, not independently reviewed; gold v2 applies the accepted disputes.
Per document
| Document | v3 | v1 | Execution | Rows | Cells pass / fail / missing | Time | Model observed |
|---|---|---|---|---|---|---|---|
| CDW Government invoice domain:finance · length:short · source:real |
0.977 | 0.955 | completed | 1 / 1 | 43 / 1 / 0 | 15s | claude-opus-5 |
| Pepco residential bill domain:energy · length:short · source:real |
1.000 | 1.000 | completed | 1 / 1 | 21 / 0 / 0 | 10s | claude-opus-5 |
| Customs entry summary domain:government · length:short · source:real |
0.997 | 0.995 | completed | 77 / 77 | 1,250 / 4 / 0 | 189s | claude-opus-5 |
| Mortgage closing disclosure domain:real estate · length:short · source:real |
0.984 | 0.984 | completed | 39 / 39 | 313 / 5 / 0 | 33s | claude-opus-5 |
| Chevrolet vehicle valuation domain:automotive · length:short · source:real |
0.991 | 0.991 | completed | 9 / 9 | 107 / 1 / 0 | 26s | claude-opus-5 |
| Medicaid remittance advice domain:healthcare · length:short · source:real |
0.964 | 0.964 | completed | 7 / 7 | 80 / 3 / 0 | 16s | claude-opus-5 |
| Caterpillar excavator specifications domain:supply chain · length:short · source:real |
1.000 | 1.000 | completed | 35 / 35 | 387 / 0 / 0 | 34s | claude-opus-5 |
| Byline Bancorp earnings report domain:finance · length:medium · source:real |
0.929 | 0.786 | completed | — | 13 / 1 / 0 | 25s | claude-opus-5 |
| Service list · synthetic domain:legal · length:medium · source:synthetic |
1.000 | 1.000 | completed | 250 / 250 | 1,502 / 0 / 0 | 420s | claude-opus-5 |
| Unclaimed property list · synthetic domain:government · length:long · source:synthetic |
0.032 | 0.032 | completed | 436 / 26,725 ⚠ | 1,305 / 5 / 78,867 | 308s | claude-opus-5 |
Time is CLI wall clock from process start to exit, including startup and tool calls. Usage tokens are recorded per attempt in the run records; subscription cost is not attributable and is shown as unknown rather than estimated.