PDF extraction › Document
finance · short document · 44 gold cells · real-source
CDW Government invoice
Gold v3 · adjudicated · corrected by dispute 001 · labels from llamaindex/ExtractBench · Source PDF ↗ · 1 page
| System | v3 | Execution | Rows | Pass / fail / missing / extra |
|---|---|---|---|---|
| Codex · GPT-6 Astra · medium effort | 1.000 | completed | 1 / 1 | 44 / 0 / 0 / 0 |
| Codex · GPT-5.6 Sol · medium effort | 0.977 | completed | 1 / 1 | 43 / 1 / 0 / 0 |
| Claude Code · Claude Opus 5 · medium effort | 0.977 | completed | 1 / 1 | 43 / 1 / 0 / 0 |
| Claude Code · Claude Sonnet 5 · medium effort | 0.886 | completed | 1 / 1 | 39 / 5 / 0 / 0 |
Pages

Fields · gold v3 against every system
| Field | Gold · gpt-6-astra · gpt-5.6-sol · claude-opus-5 · claude-sonnet-5 |
|---|---|
| bank_details.bank_name |
THE NORTHERN TRUST
claude-sonnet-5The Northern Trust wrong? |
| bank_details.account_number | 91057 wrong? |
| bank_details.routing_number | 071000152 wrong? |
| currency | USD wrong? |
| customer.name |
SUDBURY PUBLIC SCHOOLS
claude-sonnet-5Sudbury Public Schools wrong? |
| customer.address.street p.1 |
40 FAIRBANK RD STE C
gpt-5.6-solACCTS PAYABLE BUSINESS OFFICE 40 FAIRBANK RD STE C claude-opus-5ACCTS PAYABLE BUSINESS OFFICE 40 FAIRBANK RD STE C claude-sonnet-5Accts Payable, Business Office, 40 Fairbank Rd Ste C wrong? |
| customer.address.city |
SUDBURY
claude-sonnet-5Sudbury wrong? |
| customer.address.state | MA wrong? |
| customer.address.zip_code | 01776-5609 wrong? |
| customer.address.country | null wrong? |
| customer.phone | null wrong? |
| customer.email | null wrong? |
| customer.account_number | 1095519 wrong? |
| date | 2014-07-19 wrong? |
| discount_total | null wrong? |
| due_date | 2014-08-18 wrong? |
| invoice_number | NF67652 wrong? |
| invoice_type | standard wrong? |
| notes | null wrong? |
| order_number | FLDQ324 wrong? |
| original_invoice_number | null wrong? |
| payment_terms | Net 30 Days wrong? |
| purchase_order_number | 95055 wrong? |
| shipping | 0.0 wrong? |
| subtotal | 11989.25 wrong? |
| tax_total | 0.0 wrong? |
| total_amount | 11989.25 wrong? |
| vendor.name | CDW Government wrong? |
| vendor.address.street | 75 Remittance Drive, Suite 1515 wrong? |
| vendor.address.city | Chicago wrong? |
| vendor.address.state | IL wrong? |
| vendor.address.zip_code | 60675-1515 wrong? |
| vendor.address.country | null wrong? |
| vendor.phone p.1 | null fixed by dispute 001wrong? |
| vendor.email p.1 |
gachremittance@cdw.com
claude-sonnet-5null wrong? |
| vendor.tax_id | 36-4230110 wrong? |
line_items 1 row
| # | line_no | item_code | description | quantity | unit_price | discount_percent | tax_rate | amount |
|---|---|---|---|---|---|---|---|---|
| 0 | 1 wrong? | 2856676 wrong? | ACAD GOOGLE CHROME OS MGT LIC+SUP 5Y Manufacturer Part Number: CROSSWDNEDU Electronic distribution - NO MEDIA wrong? | 455.0 wrong? | 26.35 wrong? | null wrong? | null wrong? | 11989.25 wrong? |
Values are shown after the scorer's normalization. Green: matches gold under the pinned scorer, including per-field normalizers and accepted alternates. Red: differs. Grey: absent on one side.
Disputes on this document
Invoice vendor phone: account manager's direct line versus null · accepted · fields vendor.phone
Extraction schema sent to every system · 14 descriptions contain a literal gold value
The upstream schema descriptions include examples that equal this document's answers in 14 places. All systems received the same schema.
{
"$defs": {
"address": {
"description": "A postal address parsed into components.",
"properties": {
"city": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "City. Null if not present."
},
"country": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Country. Null if not printed (assume domestic; do not invent a country that is not shown)."
},
"state": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "State, province, or region (e.g. \u0027MA\u0027, \u0027TX\u0027). Do NOT confuse with country. Null if not applicable/printed."
},
"street": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Street line(s) of the address as printed (incl suite/PO box). Null if not present."
},
"zip_code": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Postal/ZIP code as printed, verbatim string incl any +4 (e.g. \u002760675-1515\u0027). Null if not present."
}
},
"type": "object"
}
},
"properties": {
"bank_details": {
"anyOf": [
{
"properties": {
"account_number": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The vendor\u0027s bank account number for payment, verbatim string. Do NOT confuse with the customer\u0027s account_number or the invoice/order numbers. Null if not printed."
},
"bank_name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Name of the bank for remittance (e.g. \u0027The Northern Trust\u0027, \u0027JPMorgan Chase\u0027). Null if not printed."
},
"routing_number": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The bank routing / ABA number for payment, verbatim string. Labels: \u0027Routing No\u0027, \u0027Bank Routing No\u0027. Null if not printed."
}
},
"type": "object"
},
{
"type": "null"
}
],
"description": "Vendor bank/remittance details for electronic payment, if an ACH/wire/routing block is present. Null if no bank details are printed on the invoice."
},
"currency": {
"description": "The ISO 4217 currency code of the invoice amounts. Read from an explicit \u0027Currency\u0027 field or a printed code (e.g. \u0027USD\u0027); otherwise infer from the currency symbol (\u0027$\u0027 on a US vendor invoice -\u003e \u0027USD\u0027). Do NOT guess beyond what the symbol/locale supports.",
"enum": [
"USD",
"EUR",
"GBP",
"CAD",
"AUD",
"JPY",
"CNY",
"INR",
"BRL",
"CHF",
"KRW"
],
"type": "string"
},
"customer": {
"description": "The party being billed (the \u0027Bill To\u0027 / \u0027Invoice To\u0027 recipient who owes payment, e.g. \u0027Hingham Town Hall Selectmen\u0027, \u0027City of Mission\u0027). Do NOT populate from the vendor letterhead/remit-to block, nor from the \u0027Ship To\u0027 / delivery address if a distinct Bill To exists.",
"properties": {
"account_number": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The customer\u0027s account number with this vendor. Labels: \u0027Customer Number\u0027, \u0027Account Number\u0027, \u0027GRAINGER ACCOUNT NUMBER\u0027, \u0027Cust No\u0027, \u0027Invoice Account\u0027. Verbatim string incl leading zeros (e.g. \u00271095519\u0027, \u0027C2588144\u0027, \u0027838233708\u0027, \u002743895\u0027, \u0027141848\u0027). Do NOT confuse with the Invoice Number, the Purchase Order Number, or the Order Number. Located in the header/Bill To block. Null if no customer/account number is printed."
},
"address": {
"$ref": "#/$defs/address"
},
"email": {
"anyOf": [
{
"format": "email",
"type": "string"
},
{
"type": "null"
}
],
"description": "Customer email as printed. Do NOT confuse with the vendor\u0027s remittance/AR email. Null if not present."
},
"name": {
"description": "The customer/Bill To name exactly as printed. Do NOT use the vendor name or the Ship To name when a distinct Bill To exists.",
"type": "string"
},
"phone": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Customer phone number as printed. Do NOT confuse with the vendor\u0027s phone. Null if not present in the Bill To block."
}
},
"type": "object"
},
"date": {
"description": "The invoice issue date, normalized to YYYY-MM-DD. Labels: \u0027Invoice Date\u0027, \u0027Date\u0027, \u0027DATE\u0027. Do NOT confuse with the Due Date, the Order Date, a per-line Tax Date / Shipment Date, or \u0027Samples Received On\u0027. Located in the header block near the invoice number. Convert from the printed format (e.g. \u002707/19/14\u0027 -\u003e \u00272014-07-19\u0027, \u002701-May-24\u0027 -\u003e \u00272024-05-01\u0027).",
"type": "string"
},
"discount_total": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"description": "Total invoice-level discount amount as a number. Labels: \u0027Discount\u0027, \u0027Total Discount\u0027. Do NOT confuse with a per-line discount_percent or with negative account-summary credit rows. Null if no invoice-level discount line is printed."
},
"due_date": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Payment due date, normalized to YYYY-MM-DD. Labels: \u0027Due Date\u0027, \u0027Payment Due\u0027, \u0027DUE DATE\u0027. Do NOT confuse with the invoice issue date or the payment-terms string (e.g. \u0027Net 30\u0027). Located in the header or terms block. Null if no explicit due date is printed."
},
"invoice_number": {
"description": "The vendor\u0027s unique identifier for THIS invoice document. Labels: \u0027Invoice Number\u0027, \u0027Invoice #\u0027, \u0027Invoice No\u0027, \u0027Invoice ID\u0027, \u0027INVOICE NUMBER\u0027. Verbatim string including any leading zeros, letters, or dashes (e.g. \u0027NF67652\u0027, \u00279823373304\u0027, \u0027025-479686\u0027, \u0027INUS246402\u0027, \u00272105629\u0027). Do NOT confuse with the Customer/Account Number, the Purchase Order Number, the Order/Sales Order Number, or any product Item Number. Located in the invoice header block, usually top-right near the date.",
"type": "string"
},
"invoice_type": {
"description": "Classification of the document: \u0027standard\u0027 for an ordinary bill/invoice; \u0027credit_note\u0027 only if the document is explicitly a credit memo / credit note (negative balance owed back to the customer); \u0027proforma\u0027 only if explicitly labeled \u0027Pro Forma\u0027. Default to \u0027standard\u0027 unless the document text clearly states otherwise. Do NOT infer \u0027credit_note\u0027 from individual negative line/credit rows inside a roll-forward account summary.",
"enum": [
"standard",
"credit_note",
"proforma"
],
"type": "string"
},
"line_items": {
"description": "The billed line items from the MAIN charges/details table ONLY (e.g. the \u0027New charges (details below)\u0027 table, the per-item \u0027Detailed Invoice Information\u0027 table, or the numbered \u0027Line No.\u0027 table). EXCLUDE: rows from an \u0027Account Summary\u0027/balance roll-forward block (Balance Forward, payments/credits, \u0027Other invoices from this date\u0027, Total Amount Due), remittance-stub rows, and subtotal/tax/total summary rows. For invoices that group charges under CATEGORY headers where the price is printed at the header level (e.g. Tyler \u0027ERP Pro Financials Annual Fees\u0027), emit ONE line_item per priced category header row (description = the category header text, amount = its printed Extended Price); do NOT emit separate rows for the unpriced indented component/\u0027Cycle\u0027 sublines.",
"items": {
"properties": {
"amount": {
"description": "The extended line amount (quantity times unit price, after any line discount, before invoice-level tax) as a number; strip currency symbols and separators. Labels: \u0027Amount\u0027, \u0027Total\u0027, \u0027EXT PRICE\u0027, \u0027Extended Price\u0027, \u0027TOTAL\u0027. Do NOT confuse with unit_price (per-unit) or with the invoice subtotal/total.",
"type": "number"
},
"description": {
"description": "The product or service NAME for this line as printed in the Description column, including an inline manufacturer/part number if printed on the same item line (e.g. \u0027OFFICE DEHUMID ... MANUFACTURER # FFAP5033W1\u0027). Tolerate OCR artifacts. EXCLUDE: (a) shipment/delivery/tracking metadata (Delivery#, Date, Carrier, No. of pkgs, Wt, Trk#, SHIPPED FROM); (b) Tax-Date / Shipment-Date / \u0027Cycle: Start.../End...\u0027 / service-period sublines; (c) customer-service annotations or \u0027Customer has indicated to invoice to...\u0027 / \u0027archived the PO\u0027 notes; (d) the item_code, quantity, unit_price, or amount columns. Normalize internal whitespace to single spaces. Do NOT confuse with item_code (the SKU/part-number column) or notes (free-text message block).",
"type": "string"
},
"discount_percent": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"description": "Line-level discount as a percentage number (e.g. 10 for 10%). Do NOT confuse with tax_rate or with a \u0027Rush %\u0027 surcharge. Null if no discount percent is printed for the line."
},
"item_code": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Vendor product code / SKU / part number for this line. Labels: \u0027Item Number\u0027, \u0027Item #\u0027, \u0027ITEM #\u0027, \u0027Item\u0027, \u0027MFG #\u0027 / \u0027Manufacturer Part Number\u0027. Verbatim string incl leading zeros and letters (e.g. \u00272856676\u0027, \u002761DE01\u0027, \u0027KID21005779\u0027, \u0027IQ10183\u0027, \u002772036\u0027). Do NOT confuse with line_no (the row sequence counter). Null if the line has no product code column (e.g. service-only invoices)."
},
"line_no": {
"description": "Printed row sequence number of this line item. Labels: \u0027Line No.\u0027, \u0027Line #\u0027, \u0027Item\u0027 (when it is a 1,2,3... counter). If no explicit Line No. column exists, assign 1..N in printed top-to-bottom order. Do NOT confuse with item_code (the product SKU/part number) nor with quantity.",
"type": "integer"
},
"quantity": {
"description": "Quantity billed for this line. Labels: \u0027Quantity\u0027, \u0027QTY\u0027, \u0027Units\u0027, \u0027QTY SHIP\u0027. May be fractional for hourly/weight-based services. Do NOT confuse with line_no or with unit_price.",
"type": "number"
},
"tax_rate": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"description": "Line-level tax rate as a percentage number (e.g. 8.25 for 8.25%). Do NOT confuse with discount_percent. Null if no per-line tax rate is printed."
},
"unit_price": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"description": "Price per unit for this line as a number (strip currency symbols and thousands separators; e.g. \u0027$2,340.00\u0027 -\u003e 2340.00). Labels: \u0027Unit Price\u0027, \u0027Price\u0027, \u0027Rate\u0027, \u0027Adjusted Price\u0027. Do NOT confuse with the line amount/extended price (that is the quantity-times-unit total). Null if no per-unit price is printed for the line."
}
},
"required": [
"line_no",
"description",
"amount"
],
"type": "object"
},
"type": "array"
},
"notes": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Verbatim content of an EXPLICITLY LABELED free-text block on the invoice (label one of: Notes, Comments, Message, Remarks, Special Instructions, Important Messages). EXCLUDE payment/remittance boilerplate, marketing (\u0027Go Green\u0027, paperless-billing promos), received/paid stamps, finance-charge terms, and vendor address-change notices. Null if the invoice has no such explicitly labeled notes block. Do NOT confuse with line_items.description or payment_terms. If a labeled block\u0027s content is entirely excluded categories (marketing/paperless/remittance boilerplate), return null."
},
"order_number": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The vendor-assigned order / sales order number identifying the order this invoice fulfills. Labels: \u0027Order Number\u0027, \u0027Sales Order Number\u0027, \u0027Ord No\u0027, \u0027ORDER NUMBER\u0027. Verbatim string as printed (e.g. \u00271491364305\u0027, \u0027S151787147\u0027, \u0027FLDQ324\u0027, \u0027189884\u0027). Do NOT confuse with the customer\u0027s Purchase Order Number (purchase_order_number), the Invoice Number, the Customer/Account Number, or \u0027Your Ref\u0027/\u0027Our Ref\u0027. Located in the header block. Null if no order/sales-order number is printed."
},
"original_invoice_number": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "For credit notes only, the invoice number of the original invoice being credited (verbatim string). Do NOT confuse with this document\u0027s own invoice_number. Null for standard and proforma invoices, or whenever no original invoice is referenced."
},
"payment_terms": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Payment terms text as printed. Labels: \u0027Terms\u0027, \u0027Payment Terms\u0027, \u0027Payment\u0027. Verbatim string (e.g. \u0027Net 30 Days\u0027, \u0027NET 45\u0027, \u0027NET30\u0027). Do NOT confuse with the due_date (a date). Null if no terms text is printed."
},
"purchase_order_number": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The customer\u0027s Purchase Order number referenced on the invoice. Labels: \u0027Purchase Order Number\u0027, \u0027PO Number\u0027, \u0027P.O. Number\u0027, \u0027PO NUMBER\u0027. Verbatim string exactly as printed (may be alphanumeric or even a free-text reference, e.g. \u002795055\u0027, \u0027212706\u0027, \u0027WEB2454487473\u0027, \u0027Dish 160R Upton St\u0027). Do NOT confuse with the vendor\u0027s Order/Sales Order Number (order_number), the Invoice Number, the Customer/Account Number, or \u0027Your Ref\u0027/\u0027Our Ref\u0027. Located in the header/terms block. Null if no PO field is present or it is blank."
},
"shipping": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"description": "Shipping / freight charge total as a number. Labels: \u0027Shipping\u0027, \u0027Freight\u0027. Do NOT confuse with the subtotal or tax_total. Null if no shipping line is printed."
},
"subtotal": {
"description": "Sum of all line item amounts before invoice-level tax, shipping, and discounts, as a number (strip currency symbols/separators). Labels: \u0027Subtotal\u0027, \u0027SUBTOTAL\u0027, \u0027INVOICE SUB TOTAL\u0027, \u0027Sub Total\u0027, \u0027Sales Amount\u0027. Do NOT confuse with total_amount (the final amount due) or with a per-line amount. Located in the totals block.",
"type": "number"
},
"tax_total": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"description": "Total tax amount on the invoice as a number. Labels: \u0027Sales Tax\u0027, \u0027Tax\u0027, \u0027TAX \u0026 BOTTLE DEPOSITS TOTAL\u0027. Do NOT confuse with a per-line tax_rate (a percentage) or with the subtotal. Null if no tax total line is printed."
},
"total_amount": {
"description": "The final total amount due on the invoice as a number (strip currency symbols/separators). Labels: \u0027Amount Due\u0027, \u0027Total\u0027, \u0027Invoice Total\u0027, \u0027Total Due\u0027, \u0027BALANCE DUE\u0027, \u0027Order Total\u0027. Do NOT confuse with the subtotal, with a per-line amount, or with an \u0027Account Summary\u0027 balance-forward figure. Located in the totals block, usually the last/boldest figure.",
"type": "number"
},
"vendor": {
"description": "The party issuing the invoice and to be paid (the \u0027Remit To\u0027 / letterhead seller, e.g. \u0027CDW Government\u0027, \u0027Axon Enterprise Inc.\u0027). Do NOT populate from the Bill To / customer block.",
"properties": {
"address": {
"description": "Seller/VENDOR postal address. TIEBREAK: use the seller\u0027s PRIMARY address printed in the top letterhead/header block next to the vendor name/logo. If a SEPARATE \u0027Remit To\u0027 / \u0027Please send payment to\u0027 / lockbox / \u0027Dept. \u003cacct\u003e\u0027 payment address is also printed, do NOT use it here (that is a remittance address).",
"properties": {
"city": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "City. Null if not present."
},
"country": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Country. Null if not printed (assume domestic; do not invent a country that is not shown)."
},
"state": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "State, province, or region (e.g. \u0027MA\u0027, \u0027TX\u0027). Do NOT confuse with country. Null if not applicable/printed."
},
"street": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Vendor street line from the letterhead/header block; NOT a Remit-To/lockbox \u0027Dept.\u0027 line."
},
"zip_code": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Postal/ZIP code as printed, verbatim string incl any +4 (e.g. \u002760675-1515\u0027). Null if not present."
}
},
"type": "object"
},
"email": {
"anyOf": [
{
"format": "email",
"type": "string"
},
{
"type": "null"
}
],
"description": "Vendor remittance/AR email as printed. Do NOT confuse with the customer\u0027s email. Null if not present."
},
"name": {
"description": "The vendor/seller name exactly as printed on the letterhead or remit-to block. Do NOT use the Bill To (customer) name.",
"type": "string"
},
"phone": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "Vendor phone number as printed. Do NOT confuse with the customer\u0027s phone. Null if not present."
},
"tax_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"description": "The vendor\u0027s tax identifier, verbatim string. Labels: \u0027FEIN\u0027, \u0027FEI Number\u0027, \u0027TIN\u0027, \u0027Tax ID\u0027, \u0027Federal ID #\u0027, \u0027Fed. ID No.\u0027. Do NOT confuse with the customer\u0027s account number or any invoice/order number. Null if not printed."
}
},
"type": "object"
}
},
"required": [
"invoice_number",
"invoice_type",
"date",
"vendor",
"customer",
"line_items",
"subtotal",
"total_amount",
"currency"
],
"type": "object"
}