EnterpriseSWE-Pro
Longer-horizon tasks in more complex enterprise environments, verified by running the system.
9 tasks · Tasks: Private · 4 trials each · Claude Opus 5.5 × Claude Code · Gemini 3.8 Flash × Terminus 2 · medium effort · 12 min median per trial
- 1Claude Opus 5.5Claude Code · medium effort63.9%9 tasks × 4 trials · $1.94 per trial · 2026-10-13
- 2Gemini 3.8 FlashTerminus 2 · medium effort11.1%9 tasks × 4 trials · $4.73 per trial · 2026-10-13
01Where the policy fails
Claude Opus 5.5 passes 23 of 36 scored trials; Gemini 3.8 Flash passes 4 of 36. We read every failed trial against the task statement and the case that failed. The most common failure is breaking behaviour that already worked. Next come fixes aimed at the wrong cause, then fixes that do half of what the task asks. Both models also name the defect they leave behind and ship anyway. No trial was lost to the sandbox: every run ended with the agent's own hand-off.
| Failure mode | Failed trials |
|---|---|
| Broke behaviour that already workedDropped the plan-version control break to stop a duplicate flat credit; repriced percentage promotions merged onto one line. | 17 |
| Fixed the wrong causeStamped each credit with the plan version of the account's last call, not the version that granted it. | 12 |
| Did half of what the task asksFlipped the credit's sign but kept the first ladder month's amount, so a month-3 account got month 1's credit. | 6 |
| Left a stated requirement unmetUpserted the rerun and kept the earlier run's stale rows, where the task asks for a single clean run. | 6 |
| Fitted the fix to the sample dataMade promotion 201 enrolments open-ended so one sample call after the enrolment ended billed under the promotion. | 4 |
Both models name the defect they leave behind
In 4 failed trials the hand-off or the reasoning states the exact case the verifier later fails. Opus files it as a limitation or a "decision for you"; Flash talks itself past it. Of the 6 failed rerun trials, none deletes the period's rows before reloading. Flash reads the DDREDE count ("how many of the rows it wrote were already there") as ruling out a delete. Both passing Opus trials count the existing rows first, then delete and load.
I didn't add an automatic purge because I couldn't scope it safely.
if two versions of a plan ever carry different monthly amounts, an account that has calls under both versions in the same month now gets the latest version's amount, once.
NO! Because if you deleted them before writing, 0 of the rows it wrote would be already there!
**Count:** before loading, the program reads the input once, using the same grouping the load uses, and counts the daily and monthly rows it is about to write that already exist.
The sample data passes; the cases do not
The agents test on the dev data the task ships with, and that data does not show the regression. So "unchanged" is declared from a diff that could not show a change. Opus broke one pass-to-pass guard on set-amount (33 of 34 cases) after claiming the percentage credits were byte-identical. On tcl-consumption, two Opus trials concluded the sample had no defect, though one account visibly leaves with another account's 64 days. All 8 trials fail the same 4 cases there.
The percentage credits come out byte-for-byte the same as before.
Every account in this sample was already correct, so the defect doesn't show in it.
correctly applying the customer's plan discount to TIM mobile calls while leaving all other traffic untouched.
Flash copies the sibling program
The estate holds near-copies of each program, and Flash copies their pattern without asking whether it fits. On mobile-network, all 4 Flash trials copied the sibling's origin-and-destination check, which re-rated calls to TIM phones as well as calls from them. All 4 Opus trials changed the origin check alone, matching the reference. On credit-ccm, all 4 Flash trials stamped the plan version of the account's last call, not the version that granted the credit.
In SIU05420.cbl and SIU05110.cbl, 'OR '8'' was already added to TERORIG and TERDEST checks.
Wait! Could we also set `WAX-CCM-CRED` there?
Flash fits the rule to the sample
On promotion-201, all 4 Flash trials read a sample call made after the enrolment ended as the complaint. They made 201 enrolments open-ended, and one hardcoded the sample's date into the routing rule. All 4 Opus trials kept that call on the base plan and fixed the load for every promotion.
What if `SIU05730` sets `WTB-DT-FIM-PLPROM` to `99991231` for promotion 201?!
(E01-DATACHA GREATER 20031231)
Five times the steps, a sixth of the passes
Opus takes a median 41 steps and 6 minutes a trial; Flash takes 212 steps and 28 minutes. Flash costs $4.73 a trial against $1.94 for Opus. On day-count, where both models pass 4 of 4, Flash took 87 to 152 steps for a fix Opus made in 20 to 30.
02The tasks
Enterprise engineering happens in context: cloud operations, DevOps, monolithic systems, large refactors, SRE. One business rule spreads across many services, jobs and configs. Each task is a change to a production system, stated as the business asks for it. A hidden verifier runs the changed system and checks the outcome.
This version runs on a monolithic mainframe billing system.
Their calls placed from TIM mobile phones come back at the standard tariff with no plan discount. The rest of their long distance is priced as it should be. The commercial team registered the plans as they were negotiated, and this traffic grows every month.
It is the middle of March and the March bills have not gone out. These calls have to bill under the customer's plan, the same as the rest of their long distance. Everything else bills as it does today, closed months stay as billed, and the close still has to fit its window.
Reprocessing has to leave the history the way a single clean run would have left it, no matter how many times we run the period, and the run's control report, DDREDE, should tell us, for the daily history and for the monthly history, how many of the rows it wrote were already there.
Please note that current invoice calculations are accurate, so actual billing output must remain unchanged.
03How the tasks are built
Tasks are authored with Taskforge on Kennet, our environment-design infrastructure. Agents draft; people direct and rule. Two principles hold throughout: the task statement is the spec, and the tests never see the reference solution.
Consult outer · a person
Rules on what no check settles: a contested case, a gap in the task statement, a verdict that disagrees with the reward.
Conduct middle · agents at k
72 trials of Claude Opus 5.5, Gemini 3.8 Flash, each trajectory read for whether the reward was earned and why a failure happened.
Check inner · in code
Each defect is in the system's own programs. The task statement gives the business outcome, never the program. Cases run the batch jobs on the emulator and compare every output dataset, table and report. Pass-to-pass cases pin what must not change. The agent's sample data is never built from the cases. Task CI runs every known-wrong fix (must fail) and every known-right fix (must pass).
- Hidden cases
- 165 (76 fail-to-pass, 89 pass-to-pass). Median 12 per task.
- Wrong arms
- 100 known-wrong solutions the verifier must reject. Median 9 per task.
- Right arms
- 14 correct solutions besides the reference that it must accept; 2 written blind from the task statement.
04Training signal
Claude Opus 5.5 Claude Code · medium effort
- 0 of 4
- 2
- 1 of 4
- 1
- 2 of 4
- 1
- 3 of 4
- 0
- 4 of 4
- 5
Gemini 3.8 Flash Terminus 2 · medium effort
- 0 of 4
- 8
- 1 of 4
- 0
- 2 of 4
- 0
- 3 of 4
- 0
- 4 of 4
- 1
- Rollout
- 12 min median agent time · $3.34 mean per trial
- Per pass
- $8.90 spent per passing trial
05Reward and method
- Environment
- Faithful emulation of batch and online mainframe systems. No network.
- Agent
- Claude Opus 5.5 on Claude Code 2.1.288; Gemini 3.8 Flash on Terminus 2 2.0.0. Medium reasoning effort for every pair. 4 trials per task.
- Verifier
- Separate container. Rebuilds the code, runs hidden tests. Pass = all tests pass.
- Score
- Share of passing trials, averaged over tasks.
- Interval
- 95% bootstrap over tasks. Shared rank unless intervals separate.