PDF extraction › System
Codex · GPT-5.6 Sol · medium effort
Mean value f1 0.989 on gold v3 (0.958 on v1) · 10 of 10 graded · 0 timeouts · median 59s · cost unknown
System
- Model
- gpt-5.6-sol
- Harness
- codex-cli 0.153.4
- Effort
- medium
- Tools
- document_info, read_page_text, read_page_image, submit_extraction
- Code execution
- yes, native code mode retained
- Timeout
- 600 s per document
- Attempts
- one per document, no retries
- Config hash
- cbd033552d5fa9cd
Track
Coding agents as extractors · restricted document reader
- Each agent received only the unchanged schema and four document tools: document info, page text, page image, submit extraction. No shell, browser, filesystem or web access.
- Codex retained its native code-mode computation and could assemble large submissions programmatically; Claude Code ran reader-only. This is a configured-agent comparison, not an equal-tool model ranking.
- One attempt per document and configuration. Timeouts at 600 seconds score zero in the fixed-denominator mean and are shown as timeouts, never hidden.
- Cost is unknown: runs used existing subscription quotas. CLI dollar estimates are not invoices.
- Source schemas contain answer hints for some fields (see the schema audit). Gold v1 is the upstream annotation, not independently reviewed; gold v2 applies the accepted disputes.
Per document
| Document | v3 | v1 | Execution | Rows | Cells pass / fail / missing | Time | Model observed |
|---|---|---|---|---|---|---|---|
| CDW Government invoice domain:finance · length:short · source:real |
0.977 | 0.955 | completed | 1 / 1 | 43 / 1 / 0 | 35s | not reported |
| Pepco residential bill domain:energy · length:short · source:real |
1.000 | 1.000 | completed | 1 / 1 | 21 / 0 / 0 | 30s | not reported |
| Customs entry summary domain:government · length:short · source:real |
0.997 | 0.995 | completed | 77 / 77 | 1,250 / 4 / 0 | 144s | not reported |
| Mortgage closing disclosure domain:real estate · length:short · source:real |
0.997 | 0.997 | completed | 39 / 39 | 317 / 1 / 0 | 90s | not reported |
| Chevrolet vehicle valuation domain:automotive · length:short · source:real |
0.991 | 0.991 | completed | 9 / 9 | 107 / 1 / 0 | 48s | not reported |
| Medicaid remittance advice domain:healthcare · length:short · source:real |
0.964 | 0.964 | completed | 7 / 7 | 80 / 3 / 0 | 62s | not reported |
| Caterpillar excavator specifications domain:supply chain · length:short · source:real |
1.000 | 1.000 | completed | 35 / 35 | 387 / 0 / 0 | 56s | not reported |
| Byline Bancorp earnings report domain:finance · length:medium · source:real |
1.000 | 0.714 | completed | — | 14 / 0 / 0 | 49s | not reported |
| Service list · synthetic domain:legal · length:medium · source:synthetic |
0.984 | 0.984 | completed | 250 / 250 | 1,478 / 24 / 0 | 440s | not reported |
| Unclaimed property list · synthetic domain:government · length:long · source:synthetic |
0.979 | 0.979 | completed | 26,983 / 26,725 ⚠ | 78,891 / 1,286 / 0 (+774 extra) | 527s | not reported |
Time is CLI wall clock from process start to exit, including startup and tool calls. Usage tokens are recorded per attempt in the run records; subscription cost is not attributable and is shown as unknown rather than estimated.