Done on the surface, wrong underneath

I ran Claude Sonnet 5.5 and Gemini 3.8 Flash once on 15 GDPval office tasks and checked the graded work against the input files. The serious failures were claims that did not match the inputs, and 9 of the 20 agent errors never lowered a rubric score.

What this study shows: Across 32 attempts on 15 GDPval tasks, I found 20 agent errors and traced each one to a failure mode and a cause. Tool misuse accounts for 7 of them, goal drift, context loss and retrieval failure for 4 each, and policy or safety for 1. Model behaviour caused 16 and the setup caused 4. What it does not show: It does not rank the models or estimate failure rates beyond this sample.
  1. Both agents stated things that were not true. 7 of the 8 high severity errors are Gemini's. Claude's text disagreed with its own cells in 3 tasks, and all three are one issue, CL-3.
  2. Rubric scores missed serious errors. The AI graders noted 19 of the 20 agent errors, yet 5 of the 8 high severity ones never lowered a score. Gemini's PO audit scored 98.8% with a credit memo that does not add up.
  3. The fix is expert records on new tasks. 16 of the 20 agent errors are model behaviour these records would target.

Examples

Four verified errors, with the evidence

Each of these looks like finished work. Pick a case to see the agent's own words, marked where the input files disagree.

XLSXWorkpaper header, quoted ↗Rubric 77.8%

Task: choose a sample of risk metrics to test from Population v2.xlsx.

Click a marked phrase to see it in the file. Rubric bars show the score the AI grader gave the attempt. Links point to one fixed commit of the public repo. Input files come from the GDPval dataset on Hugging Face.

Findings

What the agents got wrong, and what the scores missed

1. Three failure patterns

Mostly Gemini

Claims that do not match the inputs

Gemini invented facts in 3 documents and wrote 2 summaries that contradict its own cells. Claude's text disagreed with its cells in 3 tasks.

Data that targets itAn expert marks each claim as supported, assumed or invented, and links each summary to the cells behind it.

Mostly Gemini

Misstating its own work

Gemini wrote an audit sign-off that never happened and labelled formula tables as PivotTables in 2 tasks. Claude disclosed the same shortcut but gave a false reason.

Data that targets itAn expert shows what an agent may and may not claim, from approvals to substitutions.

Both models

Handling missing or conflicting inputs

Without OCR, Claude reported filled-in form fields as blank. With OCR, it flagged a missing IRS table instead of estimating it. Gemini used one of two conflicting values without a note.

Data that targets itAn expert decides what to do with missing or conflicting inputs and marks when to estimate and when to flag.

2. Agent errors by severity

Agent errors by severity and agent

HighMediumLow

There are 20 agent errors. 16 were seen in one agent and 4 in both, and an error seen in both counts once for each agent, so the bars add up to 24.

3. The AI graders noticed 19 of the 20 agent errors, but only 11 lowered a score.

What the AI grader did with each agent error

Lowered the score · 11The grader failed a rubric item for each of these, such as the missing PivotTables, the broken range mode and the misread tax forms.
Defect note only · 7The grader wrote these down as defects, which do not count toward the score. They include the invented sign-off, the credit memo that does not add up and the invented SLA.
Noticed and passed · 1The grader saw that the grill guard claim was wrong and still passed the item.
Missed · 1The grader never saw Claude's false reason for skipping PivotTables, because it was in the chat and not in the files.

I matched each agent error to the grader's rubric items and defect notes for the same attempt. High severity errors fared worst, and 5 of the 8 never lowered a score.

4. Serious failures sat behind high rubric scores

Rubric score and verified agent errors, one mark per attempt

Claude Gemini

Hover or tap a mark for the task and its errors. Counts include errors traced to one attempt. The two Gemini attempts at 0% ran out of turns and delivered nothing. The grader marked Gemini's PO audit as needing revision, but its rubric score stayed at 98.8%.

The rubric also failed correct work. The 9 scoring problems in the results are errors in the rubric, the grader or the dataset.

Dataset

What the dataset would contain

  • The dataset would use new expert tasks, because the GDPval gold subset is public. Each verified failure becomes one record that holds the evidence, the expert's answer and an automatic check.
  • Every claim in a deliverable gets one of three labels. A claim is supported when it traces to a cell or passage in the inputs, assumed when it does not but the text says so, and invented when it does not and the text states it as fact.
  • Practitioners write the records, and a second expert reviews each one so their agreement can be reported.
  • Evaluation and training use separate tasks. Training counts as a success if held-out tasks show fewer unsupported claims and fewer mismatches between text and cells, with no rise in needless hedging.

Next step

A small pilot on new tasks in the same three occupations would test this. Practitioners write the records and a second expert labels a sample to measure agreement. The audit tooling and the 123 checked grader results already exist. The records, the second labeller and the training test do not exist yet.

# one verified failure, ready for an evaluation set
id:        CL-2
task:      Wholesale PO audit, Order Clerks
type:      context loss, high severity
agent:     "credit $125.56 to Cedar & Cline ($94.08)
           and Granite Small Co. ($18.08)"
evidence:  94.08 + 18.08 = 112.16, not 125.56
expert:    Cedar & Cline 94.08, Granite 17.48,
           Maple Yellow 12.60, Nightbend 1.40
check:     recompute customer totals from SKU lines

Methods

How the study was run

Why this slice

I chose the slice before running anything: all 15 tasks from three GDPval occupations, 5 each from accountants and auditors, production supervisors and order clerks. I can judge whether a deliverable in these jobs is usable, which matters because I checked the agents' work by hand. The tasks also use the formats that dominate GDPval, since Excel, PDF and Word make up 441 of its 509 files. Every task comes with input files, so each claim in a deliverable can be checked against its source.

1. Setup

How the study ran, step by step

  1. 15tasksGDPval gold, 5 from each of 3 occupations
  2. 32attempts15 per agent, plus one OCR rerun of the tax task for each
  3. 446grader findings352 failed rubric items and 94 defects
  4. 123results checked
  5. 35verified issues20 agent errors, 6 conventions, 9 scoring problems

Each step counts a different thing, so the numbers are not a conversion rate.

Full setup: agents, graders, checks, severity and cost
TasksThe study used 15 GDPval gold tasks, 5 each from accounting, production supervision and order clerking. Every task has input files, and 14 of the 18 expected deliverables are workbooks.
AgentsClaude Sonnet 5.5 and Gemini 3.8 Flash each ran once per task with the same prompt, tools and offline sandbox, and an 80 turn limit. The tax return was rerun once with OCR.
GradersGrading was blinded and cross-family, so Claude Opus 5.5 graded Gemini and Gemini 4 Argon graded Claude. Each grader scored the rubric items and also logged defects outside the rubric.
ChecksI checked 92 grader results against the files, which were all 34 severe or tax-rerun findings and 58 sampled passes. Checker agents made 31 more checks.
DetectionFor each agent error I checked whether the grader failed a rubric item for it, logged it only as a defect note or missed it.
SeverityThe severity ratings are mine. High means a professional would act on a wrong number or have nothing to use, and the checker agents' rating is shown beside mine.
CostThe run ledgers record $46.95 in total. The first runs cost $43.66, which is $2.91 per task for two agents and grading, and the OCR rerun cost $3.29.

2. Verification

From grader findings to verified issues

The first three bars count different things, so the drop from one bar to the next is not a pass rate. The bottom bar breaks the 35 verified issues down by kind.

AI graders were right on the facts but counted one cause many times

34 grader findings checked by hand

These are grader results I checked against the files, so they are a different count from the 35 verified issues.

  1. 57 of 58 sampled passes were correct.
  2. 30 of 34 failure findings had the right facts: 14 correct and 16 partly correct. The 16 scored the work too harshly, and 12 of them came from one cause counted as several failed items, 11 from the tax deduction choice and 1 from the tour cost split. 3 were wrong and 1 could not be settled.
  3. The two graders were not equally strict on the same type of column label.

The 123 checked results are the start of a human-labelled set for calibrating graders.

Results

Every attempt and every verified issue

Verified issues by failure mode and agent

HighMediumLowFaded: noiseDashed: scoring problem

Each chip is one verified issue, ordered by severity. An issue seen in both agents appears in both columns. Select a chip to open its record.

Browse all 35 issues by researcher value or by task

Which issues matter to a researcher

I labelled 9 model errors as silent failures because they pass a quick read of the deliverable, and the graders lowered a score for only 3 of them. I labelled 5 as routine because they show in the deliverable itself. Noise is a setup effect, a harmless slip or a valid expert alternative, and each record says which. Scoring problems, shown dashed, are errors in the rubric or the grader.

All 35 verified issues, by value to a researcher

Agent Severity
Claude GeminiHighMediumLowGrader shows what the AI grader did with each agent error.

Select an issue to open its record.

Verified issues by task

HighMediumLowDashed: scoring problemFaded: noise

Expand an occupation to see each task. The grey line under each cell gives the strict rubric score, turns used and the grader's usability call. Tax return chips include the OCR rerun.

Limits

What this study cannot tell you

Runs
Each agent ran each task once, so one attempt may not show how an agent usually performs. The two models also sit in different price tiers, so the results do not rank them.
Task mix
The 15 tasks use workbooks, PDFs and Word documents, plus one text file among the inputs. Their 18 expected deliverables are 14 Excel workbooks, 2 PDFs and 2 Word documents, and their 57 input files include 33 workbooks, 20 PDFs and 3 Word documents. None of them involves PowerPoint decks, images, video, code or zip files, which other GDPval tasks do.
Coverage
Across all of GDPval, 85 of the 248 deliverables are PDFs, 65 are workbooks, 64 are Word documents and 17 are PowerPoint decks. Workbooks make up a larger share of the 15 tasks than of GDPval as a whole.
Labelling
One person labelled the failures, and no second expert reviewed them. Checker agents made 31 of the 123 checks, and I spot-checked their work.
Grading
Each grader scored only one agent, so no grader saw both agents' work. On 12 of the 30 first-run attempts, the grader saw only part of a file because of a 200,000-character limit.
Setup
Four of the 20 agent errors come from the test setup and are tagged in the records. RF-1 started because the first tax run had no OCR, but the false report of blank fields was still the model's own error.
Exposure
The tasks are public, and I did not test whether the models had seen them before.
Not measured
I did not measure how long an expert takes to write each record, and I did not test whether training on these records changes model behaviour.