PDF extraction › System
Codex · GPT-6 Astra · medium effort
Mean value f1 0.894 on gold v3 (0.863 on v1) · 9 of 10 graded · 1 timeout · median 43s · cost unknown
System
- Model
- gpt-6-astra
- Harness
- codex-cli 0.153.4
- Effort
- medium
- Tools
- document_info, read_page_text, read_page_image, submit_extraction
- Code execution
- yes, native code mode retained
- Timeout
- 600 s per document
- Attempts
- one per document, no retries
- Config hash
- 9e6f145769028e67
Track
Coding agents as extractors · restricted document reader
- Each agent received only the unchanged schema and four document tools: document info, page text, page image, submit extraction. No shell, browser, filesystem or web access.
- Codex retained its native code-mode computation and could assemble large submissions programmatically; Claude Code ran reader-only. This is a configured-agent comparison, not an equal-tool model ranking.
- One attempt per document and configuration. Timeouts at 600 seconds score zero in the fixed-denominator mean and are shown as timeouts, never hidden.
- Cost is unknown: runs used existing subscription quotas. CLI dollar estimates are not invoices.
- Source schemas contain answer hints for some fields (see the schema audit). Gold v1 is the upstream annotation, not independently reviewed; gold v2 applies the accepted disputes.
Per document
| Document | v3 | v1 | Execution | Rows | Cells pass / fail / missing | Time | Model observed |
|---|---|---|---|---|---|---|---|
| CDW Government invoice domain:finance · length:short · source:real |
1.000 | 0.977 | completed | 1 / 1 | 44 / 0 / 0 | 34s | not reported |
| Pepco residential bill domain:energy · length:short · source:real |
1.000 | 1.000 | completed | 1 / 1 | 21 / 0 / 0 | 30s | not reported |
| Customs entry summary domain:government · length:short · source:real |
0.997 | 0.995 | completed | 77 / 77 | 1,250 / 4 / 0 | 121s | not reported |
| Mortgage closing disclosure domain:real estate · length:short · source:real |
0.984 | 0.984 | completed | 39 / 39 | 313 / 5 / 0 | 70s | not reported |
| Chevrolet vehicle valuation domain:automotive · length:short · source:real |
1.000 | 1.000 | completed | 9 / 9 | 108 / 0 / 0 | 52s | not reported |
| Medicaid remittance advice domain:healthcare · length:short · source:real |
0.964 | 0.964 | completed | 7 / 7 | 80 / 3 / 0 | 35s | not reported |
| Caterpillar excavator specifications domain:supply chain · length:short · source:real |
0.992 | 0.992 | completed | 35 / 35 | 384 / 3 / 0 | 43s | not reported |
| Byline Bancorp earnings report domain:finance · length:medium · source:real |
1.000 | 0.714 | completed | — | 14 / 0 / 0 | 40s | not reported |
| Service list · synthetic domain:legal · length:medium · source:synthetic |
1.000 | 1.000 | completed | 250 / 250 | 1,502 / 0 / 0 | 105s | not reported |
| Unclaimed property list · synthetic domain:government · length:long · source:synthetic |
0.000 | 0.000 | timeout | — | — | 600s | not reported |
Time is CLI wall clock from process start to exit, including startup and tool calls. Usage tokens are recorded per attempt in the run records; subscription cost is not attributable and is shown as unknown rather than estimated.