Benchmax

PDF extraction › Document

supply chain · short document · 387 gold cells · real-source

Caterpillar excavator specifications

Gold v3 · imported · labels from llamaindex/ExtractBench · Source PDF ↗ · 1 page

Systemv3ExecutionRowsPass / fail / missing / extra
Codex · GPT-6 Astra · medium effort 0.992 completed 35 / 35 384 / 3 / 0 / 0
Codex · GPT-5.6 Sol · medium effort 1.000 completed 35 / 35 387 / 0 / 0 / 0
Claude Code · Claude Opus 5 · medium effort 1.000 completed 35 / 35 387 / 0 / 0 / 0
Claude Code · Claude Sonnet 5 · medium effort 1.000 completed 35 / 35 387 / 0 / 0 / 0

Pages

Caterpillar excavator specifications page 1
Page 1

Fields · gold v3 against every system

FieldGold · gpt-6-astra · gpt-5.6-sol · claude-opus-5 · claude-sonnet-5
manufacturer Caterpillar wrong?
product_title 312C L Hydraulic Excavators wrong?

specifications 35 rows · differences

#sectionsymbolnameconditionsvaluevalue_imperialmintypmaxunitunit_imperial
0 Engine wrong? null wrong? Engine Model wrong? null wrong? Cat 3064 T wrong? null wrong? null wrong? null wrong? null wrong? null wrong? null wrong?
1 Engine wrong? null wrong? Gross Power wrong? null wrong? 70 wrong? 94 wrong? null wrong? null wrong? null wrong? kW wrong? hp wrong?
2 Engine wrong? null wrong? Flywheel Power wrong? null wrong? 67 wrong? 90 wrong? null wrong? null wrong? null wrong? kW wrong? hp wrong?
3 Engine wrong? null wrong? ISO 9249 wrong? null wrong? 67 wrong? 90 wrong? null wrong? null wrong? null wrong? kW wrong? hp wrong?
4 Engine wrong? null wrong? SAE J1349 wrong? null wrong? 67 wrong? 90 wrong? null wrong? null wrong? null wrong? kW wrong? hp wrong?
5 Engine wrong? null wrong? EEC 80/1269 wrong? null wrong? 67 wrong? 90 wrong? null wrong? null wrong? null wrong? kW wrong? hp wrong?
6 Drive wrong? null wrong? Travel Speed wrong? null wrong? 5.5 wrong? 3.4 wrong? null wrong? null wrong? null wrong? kph
gpt-6-astrakm/h
wrong?
mph wrong?
7 Engine wrong? null wrong? Bore wrong? null wrong? 102 wrong? 4 wrong? null wrong? null wrong? null wrong? mm wrong? in wrong?
8 Drive wrong? null wrong? Max. Drawbar Pull wrong? null wrong? 110 wrong? 24,720 wrong? null wrong? null wrong? null wrong? kN wrong? lb wrong?
9 Hydraulic System wrong? null wrong? Main Implement System - Max. Flow (2x) wrong? null wrong? 127 wrong? 33.5 wrong? null wrong? null wrong? null wrong? L/min wrong? gal/min wrong?
10 Engine wrong? null wrong? Stroke wrong? null wrong? 130 wrong? 5.1 wrong? null wrong? null wrong? null wrong? mm wrong? in wrong?
11 Engine wrong? null wrong? Displacement wrong? null wrong? 4.25 wrong? 259 wrong? null wrong? null wrong? null wrong? L wrong? in3 wrong?
12 Hydraulic System wrong? null wrong? Max. Pressure - Implements wrong? null wrong? 29 900 wrong? 4,340 wrong? null wrong? null wrong? null wrong? kPa wrong? psi wrong?
13 Hydraulic System wrong? null wrong? Max. Pressure - Travel wrong? null wrong? 34 300 wrong? 4,980 wrong? null wrong? null wrong? null wrong? kPa wrong? psi wrong?
14 Hydraulic System wrong? null wrong? Max Pressure - Swing wrong? null wrong? 23 050 wrong? 3,340 wrong? null wrong? null wrong? null wrong? kPa wrong? psi wrong?
15 Hydraulic System wrong? null wrong? Pilot System - Max. Flow wrong? null wrong? 23.7 wrong? 6.3 wrong? null wrong? null wrong? null wrong? L/min wrong? gal/min wrong?
16 Hydraulic System wrong? null wrong? Pilot System - Max. Pressure wrong? null wrong? 4120 wrong? 600 wrong? null wrong? null wrong? null wrong? kPa wrong? psi wrong?
17 Hydraulic System wrong? null wrong? Blade - Max. Flow wrong? null wrong? 58.5 wrong? 15.5 wrong? null wrong? null wrong? null wrong? L/min wrong? gal/min wrong?
18 Hydraulic System wrong? null wrong? Blade System - Max Pressure wrong? null wrong? 20 594 wrong? 2,990 wrong? null wrong? null wrong? null wrong? kPa wrong? psi wrong?
19 Weights wrong? null wrong? Operating Weight wrong? null wrong? 13 140 wrong? 28,970 wrong? null wrong? null wrong? null wrong? kg wrong? lb wrong?
20 Hydraulic System wrong? null wrong? Boom Cylinder - Bore wrong? null wrong? 110 wrong? 4.3 wrong? null wrong? null wrong? null wrong? mm wrong? in wrong?
21 Swing Mechanism wrong? null wrong? Swing Torque wrong? null wrong? 30 500 wrong? 22,496 wrong? null wrong? null wrong? null wrong? N·m wrong? lb ft
gpt-6-astralbf-ft
wrong?
22 Hydraulic System wrong? null wrong? Boom Cylinder - Stroke wrong? null wrong? 1015 wrong? 40 wrong? null wrong? null wrong? null wrong? mm wrong? in wrong?
23 Swing Mechanism wrong? null wrong? Swing Speed wrong? null wrong? 12.9 wrong? null wrong? null wrong? null wrong? null wrong? RPM
gpt-6-astrarpm
wrong?
null wrong?
24 Hydraulic System wrong? null wrong? Stick Cylinder - Bore wrong? null wrong? 120 wrong? 4.7 wrong? null wrong? null wrong? null wrong? mm wrong? in wrong?
25 Hydraulic System wrong? null wrong? Stick Cylinder - Stroke wrong? null wrong? 1197 wrong? 47.1 wrong? null wrong? null wrong? null wrong? mm wrong? in wrong?
26 Hydraulic System wrong? null wrong? Bucket Cylinder - Bore wrong? null wrong? 100 wrong? 3.9 wrong? null wrong? null wrong? null wrong? mm wrong? in wrong?
27 Hydraulic System wrong? null wrong? Bucket Cylinder - Stroke wrong? null wrong? 939 wrong? 37 wrong? null wrong? null wrong? null wrong? mm wrong? in wrong?
28 Service Refill Capacities wrong? null wrong? Fuel Tank wrong? null wrong? 250 wrong? 66 wrong? null wrong? null wrong? null wrong? L wrong? gal wrong?
29 Service Refill Capacities wrong? null wrong? Cooling System wrong? null wrong? 17.5 wrong? 4.6 wrong? null wrong? null wrong? null wrong? L wrong? gal wrong?
30 Service Refill Capacities wrong? null wrong? Engine Oil wrong? null wrong? 17.5 wrong? 4.6 wrong? null wrong? null wrong? null wrong? L wrong? gal wrong?
31 Service Refill Capacities wrong? null wrong? Swing Drive wrong? null wrong? 3 wrong? 0.8 wrong? null wrong? null wrong? null wrong? L wrong? gal wrong?
32 Service Refill Capacities wrong? null wrong? Final Drive (Each) wrong? null wrong? 2.5 wrong? 0.66 wrong? null wrong? null wrong? null wrong? L wrong? gal wrong?
33 Service Refill Capacities wrong? null wrong? Hydraulic System (Including Tank) wrong? null wrong? 162 wrong? 42.8 wrong? null wrong? null wrong? null wrong? L wrong? gal wrong?
34 Service Refill Capacities wrong? null wrong? Hydraulic Tank wrong? null wrong? 90 wrong? 23.8 wrong? null wrong? null wrong? null wrong? L wrong? gal wrong?

Values are shown after the scorer's normalization. Green: matches gold under the pinned scorer, including per-field normalizers and accepted alternates. Red: differs. Grey: absent on one side.

Extraction schema sent to every system · 102 descriptions contain a literal gold value

The upstream schema descriptions include examples that equal this document's answers in 102 places. All systems received the same schema.

{
  "$defs": {
    "Specification": {
      "properties": {
        "conditions": {
          "description": "The test-condition / package / device-variant qualifier printed in a separate conditions column or sub-cell on the row (electronics datasheets), e.g. \u0027CONT, RESET, THRES, TRIG\u0027, \u0027FK package\u0027, \u0027JG package, 60 seconds\u0027, \u0027NA555, NE555, SA555\u0027, \u0027SE555\u0027. When one parameter has several rows that differ only by this qualifier (e.g. \u0027Operating free-air temperature\u0027 listed once per device variant), this field is what distinguishes them. Use null when the row has no separate conditions column (Caterpillar equipment rows have no conditions column \u2014 conditions is null for them). Do NOT confuse with `name` (the parameter) or `unit`.",
          "type": [
            "string",
            "null"
          ]
        },
        "max": {
          "description": "The MAX-column value on a rated-range spec row (electronics datasheets), recorded exactly as printed including any sign e.g. \u002718\u0027, \u0027+/-225\u0027 (render \u0027\u00b1\u0027 as \u0027+/-\u0027), \u0027150\u0027, \u0027300\u0027, \u002716\u0027, \u0027+/-200\u0027, \u0027+/-1500\u0027. Use null when the MAX cell is blank or the table has no MAX column (Caterpillar single-value rows -\u003e null). Do NOT confuse with `value` (a single SI value), `min`, or `value_imperial`.",
          "type": [
            "string",
            "null"
          ]
        },
        "min": {
          "description": "The MIN-column value on a rated-range spec row (electronics datasheets: Absolute Maximum Ratings / Recommended Operating Conditions / ESD tables have MIN MAX UNIT columns), recorded exactly as printed e.g. \u00274.5\u0027, \u0027-65\u0027, \u0027-40\u0027, \u00272.75\u0027. Use null when the MIN cell is blank or the table has no MIN column (Caterpillar single-value rows -\u003e null). Do NOT confuse with `value` (a single SI value), `typ`, or `max`.",
          "type": [
            "string",
            "null"
          ]
        },
        "name": {
          "description": "The descriptive parameter label for the row, recorded VERBATIM from the parameter column (e.g. \u0027Net Power - ISO 9249\u0027, \u0027Gross Power - ISO 14396/SAE J1995\u0027, \u0027Maximum Pressure - Equipment - Normal\u0027, \u0027Boom Cylinder - Bore\u0027, \u0027Supply voltage\u0027, \u0027Output current\u0027, \u0027Operating virtual junction temperature\u0027, \u0027Storage temperature\u0027). Keep qualifier text that is part of the printed label (standard name, ISO/SAE reference, \u0027Each\u0027, \u0027(each side)\u0027). STRIP trailing footnote-reference markers (superscript numbers / parenthesised footnote indices such as \u0027(1)\u0027, \u0027(2)\u0027) and trademark symbols (\u00ae, \u2122): e.g. \u0027Supply voltage(2)\u0027 -\u003e \u0027Supply voltage\u0027. Do NOT confuse with: the `section` heading above it, the `symbol` abbreviation, or the `conditions` qualifier that lives in a separate test-conditions column.",
          "type": "string"
        },
        "section": {
          "description": "The spec-group heading the row is printed under \u2014 the bold/shaded band or sub-title directly above the row. Equipment spec sheets (Caterpillar): e.g. \u0027Engine\u0027, \u0027Engine rpm\u0027, \u0027Drive\u0027, \u0027Hydraulic System\u0027, \u0027Swing Mechanism\u0027, \u0027Weights\u0027, \u0027Track\u0027, \u0027Service Refill Capacities\u0027. Electronics datasheets (TI): e.g. \u0027Absolute Maximum Ratings\u0027, \u0027ESD Ratings\u0027, \u0027Recommended Operating Conditions\u0027. Record ONLY the heading words. NEVER include a leading section/clause number \u2014 strip any \u00275.1\u0027, \u00275.2\u0027, \u00275.3\u0027 (or \u00275\u0027, \u00276.2\u0027) prefix: a heading printed \u00275.1 Absolute Maximum Ratings\u0027 MUST be recorded as \u0027Absolute Maximum Ratings\u0027, and \u00275.3 Recommended Operating Conditions\u0027 as \u0027Recommended Operating Conditions\u0027. Carry the current heading down to every row beneath it until a new heading appears. Do NOT confuse with: the parameter label itself (`name`), the document title, or a column header inside the table (\u0027MIN\u0027, \u0027MAX\u0027, \u0027kg\u0027, \u0027lb\u0027).",
          "type": "string"
        },
        "symbol": {
          "description": "The short engineering symbol printed in the left-most symbol column of the row, when the table has one (electronics datasheets), e.g. \u0027VCC\u0027, \u0027VI\u0027, \u0027IO\u0027, \u0027TJ\u0027, \u0027Tstg\u0027, \u0027V(ESD)\u0027, \u0027TA\u0027. Record the symbol text as printed with subscripts inline (no LaTeX), e.g. \u0027VCC\u0027, \u0027IO\u0027, \u0027TJ\u0027, \u0027Tstg\u0027, \u0027TA\u0027. If the symbol cell is merged or blank because the row shares the symbol of the row directly above (e.g. both ESD-discharge sub-rows belong to symbol \u0027V(ESD)\u0027), CARRY THE SYMBOL DOWN and record it on every shared sub-row. Use null when the table has no symbol column (Caterpillar equipment spec blocks have no symbol column \u2014 symbol is null for every Caterpillar row). Do NOT confuse with the descriptive `name` (the symbol is the abbreviation, the name is the words) or the `unit`.",
          "type": [
            "string",
            "null"
          ]
        },
        "typ": {
          "description": "The TYP/NOM-column value on a rated-range spec row, recorded as printed. Null when there is no TYP/NOM column or the cell is blank (the in-scope Absolute Maximum / ESD / Recommended Operating Conditions tables and all Caterpillar blocks have no TYP column, so typ is null). Do NOT confuse with `min`, `max`, or `value`.",
          "type": [
            "string",
            "null"
          ]
        },
        "unit": {
          "description": "The unit of measure for this row\u0027s value(s)/min/typ/max, taken from the SI/primary unit column or the datasheet UNIT column. Render in plain ASCII with exponents inline as digits: \u0027kW\u0027, \u0027mm\u0027, \u0027L\u0027, \u0027kg\u0027, \u0027km/h\u0027, \u0027kN\u0027, \u0027kPa\u0027, \u0027N\u00b7m\u0027, \u0027rpm\u0027, \u0027V\u0027, \u0027mA\u0027, \u0027\u00b0C\u0027, \u0027in3\u0027 (NOT \u0027in\u00b3\u0027), \u0027m3\u0027 (NOT \u0027m\u00b3\u0027). For dual-unit Caterpillar rows this is the SI unit only (the imperial unit goes in `unit_imperial`). Use null when the spec is dimensionless (counts like number of rollers, ratios, model names). Do NOT confuse with `unit_imperial` (the imperial unit) or with the `symbol`.",
          "type": [
            "string",
            "null"
          ]
        },
        "unit_imperial": {
          "description": "The imperial/US-customary unit paired with `value_imperial` on a dual-unit equipment row, plain ASCII with digit exponents (e.g. \u0027hp\u0027, \u0027in\u0027, \u0027gal\u0027, \u0027lb\u0027, \u0027mph\u0027, \u0027lbf\u0027, \u0027psi\u0027, \u0027lbf-ft\u0027, \u0027in3\u0027 NOT \u0027in\u00b3\u0027). Use null when the row has no imperial value (single-value rows, dimensionless specs, or electronics datasheets). Do NOT confuse with `unit` (the SI unit).",
          "type": [
            "string",
            "null"
          ]
        },
        "value": {
          "description": "The single printed measurement value for a single-value spec row, in the PRIMARY/SI unit column (Caterpillar equipment blocks list one SI value, e.g. \u0027194\u0027, \u0027105\u0027, \u00277.01\u0027, \u002711.50\u0027, \u002730 900\u0027, \u002750\u0027, \u00271,900\u0027). Record ONLY the leading primary number exactly as printed (keep an internal space or comma in a grouped number like \u002730 900\u0027); DROP the unit (it goes in `unit`), DROP any parenthetical breakdown that follows the number (e.g. \u0027560 (280 \u00d7 2 pumps)\u0027 -\u003e \u0027560\u0027; \u002735\u00b0/70%\u0027 -\u003e \u002735\u00b0/70%\u0027 is kept since it is the whole value), and STRIP trademark symbols \u00ae and \u2122 (e.g. engine model \u0027Cat\u00ae C7.1\u0027 -\u003e \u0027Cat C7.1\u0027). For a non-numeric spec the descriptive value goes here (e.g. engine model \u0027Cat C7.1\u0027, verbatim minus \u00ae). ALSO use `value` (NOT min/max) for an electronics-datasheet table that has a single \u0027VALUE\u0027 column instead of MIN/MAX columns \u2014 e.g. the \u0027ESD Ratings\u0027 table, where the rated ESD voltage goes in `value` formatted with a leading \u0027+/-\u0027 for the \u00b1 sign (e.g. \u0027+/-500\u0027, \u0027+/-1500\u0027); leave min/typ/max null on those rows. Use null on rows that instead report a rated range via MIN/TYP/MAX columns (Absolute Maximum Ratings / Recommended Operating Conditions \u2014 leave value null and fill min/typ/max). Do NOT confuse with `value_imperial` (the second, imperial value) or with min/max.",
          "type": [
            "string",
            "null"
          ]
        },
        "value_imperial": {
          "description": "The second value on a dual-unit equipment spec row \u2014 the imperial/US-customary value printed to the right of the SI value (e.g. SI \u0027194 kW\u0027 -\u003e imperial \u0027260\u0027 hp; SI \u0027105 mm\u0027 -\u003e \u00274\u0027 in; SI \u0027474 L\u0027 -\u003e \u0027125.2\u0027 gal; SI \u002730 900 kg\u0027 -\u003e \u002768,200\u0027 lb; SI \u00275.3 km/h\u0027 -\u003e \u00273.3\u0027 mph). Record only the leading number as printed without its unit (the imperial unit goes in `unit_imperial`); drop parenthetical breakdowns. Use null when the row prints only one value, when the spec is dimensionless (counts, ratios, model names), or for electronics datasheets (single-unit -\u003e null). Do NOT confuse with `value` (the SI value) or with `max`.",
          "type": [
            "string",
            "null"
          ]
        }
      },
      "required": [
        "section",
        "name"
      ],
      "type": "object"
    }
  },
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "manufacturer": {
      "description": "The company that makes the product, as the full company name. Normalize a stylized brand logo to the full legal/common company name: the Caterpillar \u0027CAT\u0027 logo -\u003e \u0027Caterpillar\u0027; \u0027Texas Instruments\u0027 -\u003e \u0027Texas Instruments\u0027. Always output \u0027Caterpillar\u0027 (never \u0027CAT\u0027 or \u0027Cat\u0027) for Caterpillar documents. Do NOT confuse with a standards body named in a spec row (ISO, SAE, ANSI, JEDEC) or with the product model.",
      "type": "string"
    },
    "product_title": {
      "description": "The product\u0027s headline title as printed at the top of the spec sheet / datasheet, VERBATIM (e.g. \u0027330 Hydraulic Excavator\u0027, \u0027312C L Hydraulic Excavators\u0027, \u0027NA555, NE555, SA555, SE555\u0027). EXCLUDE trailing running-header words that describe the document rather than the product \u2014 drop a trailing \u0027specifications\u0027 / \u0027Specifications\u0027 / \u0027Technical Specifications\u0027 / \u0027datasheet\u0027 (e.g. running header \u0027312C L Hydraulic Excavators specifications\u0027 -\u003e \u0027312C L Hydraulic Excavators\u0027). For a datasheet covering several part numbers, record the full part-number list exactly as printed in the running title. Do NOT confuse with the `manufacturer`, a `section` heading, or an individual `name` row.",
      "type": "string"
    },
    "specifications": {
      "description": "Every rated-specification row in the in-scope spec tables of the document \u2014 one entry per data row. INCLUDE every parameter row under the equipment \u0027Technical Specifications\u0027 / \u0027specifications\u0027 blocks (Engine, Engine rpm, Drive, Hydraulic System, Swing Mechanism, Weights, Track, Service Refill Capacities, etc.) and, for electronics datasheets, every row of the \u0027Absolute Maximum Ratings\u0027, \u0027ESD Ratings\u0027, and \u0027Recommended Operating Conditions\u0027 tables. EXCLUDE: the \u0027Standards\u0027 section (standards-compliance reference lists like \u0027Cab/FOGS: SAE J1356 / ISO 10262\u0027) and the \u0027Sound Performance\u0027 section (sound-level prose/notes); prose bullet notes and footnotes; figure/dimension diagrams; multi-column configuration-comparison matrices (operating-weights-by-track-width, dimensions-by-stick, working-ranges); and (for datasheets) the per-condition Electrical / Switching / Thermal characteristics tables. Skip pure heading rows and blank rows.",
      "items": {
        "$ref": "#/$defs/Specification"
      },
      "type": "array"
    }
  },
  "required": [
    "manufacturer",
    "product_title",
    "specifications"
  ],
  "title": "Product Spec Sheet / Datasheet \u2014 rated specifications table (vendor-agnostic)",
  "type": "object"
}