PDF extraction › System
Claude Code · Claude Sonnet 5 · medium effort
Mean value f1 0.881 on gold v3 (0.852 on v1) · 9 of 10 graded · 1 timeout · median 38s · cost unknown
System
- Model
- claude-sonnet-5
- Harness
- 2.1.263 (Claude Code)
- Effort
- medium
- Tools
- document_info, read_page_text, read_page_image, submit_extraction
- Code execution
- no, reader-only
- Timeout
- 600 s per document
- Attempts
- one per document, no retries
- Config hash
- 1fac49d56e4bc709
Track
Coding agents as extractors · restricted document reader
- Each agent received only the unchanged schema and four document tools: document info, page text, page image, submit extraction. No shell, browser, filesystem or web access.
- Codex retained its native code-mode computation and could assemble large submissions programmatically; Claude Code ran reader-only. This is a configured-agent comparison, not an equal-tool model ranking.
- One attempt per document and configuration. Timeouts at 600 seconds score zero in the fixed-denominator mean and are shown as timeouts, never hidden.
- Cost is unknown: runs used existing subscription quotas. CLI dollar estimates are not invoices.
- Source schemas contain answer hints for some fields (see the schema audit). Gold v1 is the upstream annotation, not independently reviewed; gold v2 applies the accepted disputes.
Per document
| Document | v3 | v1 | Execution | Rows | Cells pass / fail / missing | Time | Model observed |
|---|---|---|---|---|---|---|---|
| CDW Government invoice domain:finance · length:short · source:real |
0.886 | 0.886 | completed | 1 / 1 | 39 / 5 / 0 | 17s | claude-sonnet-5 |
| Pepco residential bill domain:energy · length:short · source:real |
1.000 | 1.000 | completed | 1 / 1 | 21 / 0 / 0 | 11s | claude-sonnet-5 |
| Customs entry summary domain:government · length:short · source:real |
0.997 | 0.995 | completed | 77 / 77 | 1,250 / 4 / 0 | 151s | claude-sonnet-5 |
| Mortgage closing disclosure domain:real estate · length:short · source:real |
0.962 | 0.962 | completed | 39 / 39 | 306 / 12 / 0 | 41s | claude-sonnet-5 |
| Chevrolet vehicle valuation domain:automotive · length:short · source:real |
1.000 | 1.000 | completed | 9 / 9 | 108 / 0 / 0 | 33s | claude-sonnet-5 |
| Medicaid remittance advice domain:healthcare · length:short · source:real |
0.964 | 0.964 | completed | 7 / 7 | 80 / 3 / 0 | 59s | claude-sonnet-5 |
| Caterpillar excavator specifications domain:supply chain · length:short · source:real |
1.000 | 1.000 | completed | 35 / 35 | 387 / 0 / 0 | 32s | claude-sonnet-5 |
| Byline Bancorp earnings report domain:finance · length:medium · source:real |
1.000 | 0.714 | completed | — | 14 / 0 / 0 | 38s | claude-sonnet-5 |
| Service list · synthetic domain:legal · length:medium · source:synthetic |
0.999 | 0.999 | completed | 250 / 250 | 1,500 / 2 / 0 | 348s | claude-sonnet-5 |
| Unclaimed property list · synthetic domain:government · length:long · source:synthetic |
0.000 | 0.000 | timeout | — | — | 600s | claude-sonnet-5 |
Time is CLI wall clock from process start to exit, including startup and tool calls. Usage tokens are recorded per attempt in the run records; subscription cost is not attributable and is shown as unknown rather than estimated.