PDF extraction › CDW Government invoice › Dispute
Dispute 001 · accepted · produces gold v2 · decided 2026-09-06
Invoice vendor phone: account manager's direct line versus null
Fields: vendor.phone. Raised by Benchmax agent-track review, AGENT_FINDINGS.md. Decided by Benchmax review (Rishabh Jain), 2026-09-06.
Gold before and after
| Field | v1 | v2 | also accepted |
|---|---|---|---|
| vendor.phone | null | null | 203-851-7119 (phone_digits) |
Effect on scores · this document
| System | before | after |
|---|---|---|
| Codex · GPT-6 Astra · medium effort | 0.977 | 1.000 |
| Codex · GPT-5.6 Sol · medium effort | 0.955 | 0.977 |
| Claude Code · Claude Opus 5 · medium effort | 0.955 | 0.977 |
| Claude Code · Claude Sonnet 5 · medium effort | 0.886 | 0.886 |
Regraded from saved outputs. No configuration was rerun.
Original gold
vendor.phone = null. Comparator case_insensitive, single evidence entry with a null value.
Evidence
Page 1, bottom-left block, printed as:
ACCOUNT MANAGER JIM PINTO 203-851-7119 jimpin@cdwg.com
No other telephone number is printed anywhere on the page. The remittance block
at the top carries an e-mail (gachremittance@cdw.com) and bank details but no
phone. The schema description for the field reads: "Vendor phone number as
printed. Do NOT confuse with the customer's phone. Null if not present."
All three configurations that submitted this document (Codex Astra, Codex Sol,
Claude Opus 5) returned 203-851-7119. Claude Sonnet 5 also returned it.
Decision
Accept both. expected stays null; 203-851-7119 is added as an acceptable
alternate value, and the phone_digits normalizer is enabled so formatting
variants such as (203) 851-7119 are not penalized.
Rationale
The schema defines the vendor as the issuing party and asks for the vendor's
phone "as printed", excluding only the customer's phone. An account manager's
direct line is a CDW number printed on a CDW invoice; a reader looking for "the
vendor's phone" would plausibly take it. The upstream gold treats the
remittance e-mail as the vendor e-mail, so it already accepts a
department-level contact as the vendor's contact. Reading the account
manager's line the same way is consistent with that choice. Equally, null
is defensible because no general vendor switchboard number is printed. When a
schema leaves two readings open, the benchmark should not penalize either.
The lasting fix is a schema clarification, tracked for the clean-schema
track: state whether a named employee's direct line counts as the vendor
phone.
Effect on scores
One cell of 44. Every configuration's invoice score rises from 0.9773 (Astra), 0.9545 (Sol, Opus), 0.8864 (Sonnet) by one matched cell under v2; the change is identical across configurations, so the ranking on this document is unchanged.
Source: 001-invoice-vendor-phone.md and its gold patch. To contest this decision, open a new dispute referencing it.