Takeaways
- Sonnet and Argon tie on success (85% and 80%, intervals overlap); Argon pays 9× as much per success.
- After caching, price per token is within 20% across pairs. Steps (30 to 260 a trial) and context (103k to 287k tokens a step) set the cost.
- Most of Sonnet's, Argon's and Muse's failures are on tasks they pass in another trial: inconsistency, not capability. Flash is the exception.
- No pair stops sooner when it is failing.
01Effectiveness
| Tasks | Claude Sonnet 5.5Claude Code | Gemini 4 ArgonAntigravity CLI | Muse Spark 1.3Muse Code | Gemini 3.8 FlashTerminus 2 |
|---|---|---|---|---|
| All tasks | ||||
| Success rate, pass@1 | 85% | 80% | 63% | 60% |
| Ceiling, pass@3 | 93% | 89% | 80% | 71% |
| Reliability, pass^3 | 74% | 69% | 45% | 48% |
| Difficulty | ||||
| Easy50 tasks | 91% | 83% | 84% | 84% |
| Medium59 tasks | 86% | 86% | 63% | 57% |
| Hard41 tasks | 77% | 65% | 38% | 33% |
| Systems conditions the statement demands | ||||
| Concurrent requests72 tasks | 83% | 75% | 57% | 51% |
| A process killed mid-work34 tasks | 82% | 74% | 58% | 46% |
| Several copies of a service44 tasks | 86% | 78% | 54% | 50% |
| A slow or failing dependency64 tasks | 86% | 75% | 56% | 53% |
| A stated deadline60 tasks | 81% | 76% | 54% | 54% |
| None of the five39 tasks | 90% | 83% | 74% | 67% |
| Three or more50 tasks | 86% | 75% | 54% | 47% |
| Avg rank over tasks1 is best | 2.12 | 2.22 | 2.79 | 2.86 |
0 of 41 of 42 of 43 of 44 of 4
Figure 1. Tasks by trials passed. Each task by how many of the pair's 4 trials passed.
- Claude Sonnet 5.5Claude Code528
- Gemini 4 ArgonAntigravity CLI537
- Gemini 3.8 FlashTerminus 2527
| Guards held, share of must-fix cases fixed | under 25% | 25 to 50% | 50 to 75% | 75% or more |
|---|---|---|---|---|
| Claude Sonnet 5.5 | 1 | 2 | 7 | 49 |
| Gemini 4 Argon | 0 | 0 | 4 | 34 |
| Muse Spark 1.3 | 11 | 4 | 29 | 111 |
| Gemini 3.8 Flash | 7 | 2 | 18 | 109 |
| Where failures fall | Claude Sonnet 5.5Claude Code | Gemini 4 ArgonAntigravity CLI | Muse Spark 1.3Muse Code | Gemini 3.8 FlashTerminus 2 |
|---|---|---|---|---|
| Failing trials | 88 | 123 | 222 | 242 |
| On tasks the pair never passes | 36 | 52 | 100 | 163 |
| On tasks the pair passes in another trial | 52 | 71 | 122 | 79 |
Most failures are near misses
About half of Sonnet's, Muse's and Flash's failures broke nothing and fixed three quarters or more of the must-fix cases.
Inconsistency, not capability
Over half of Sonnet's, Argon's and Muse's failures are on tasks the pair passes in another trial. Two thirds of Flash's are on tasks it never passes.
02Efficiency
Pareto frontierCheaper and more accurate than the median pair95% interval
Figure 3. Success rate against dollars per success. Ringed pairs are the Pareto frontier.
Trials per passSteps per trialTokens per stepPrice per token, after cachingTotal, the product, in the pair’s hue
Figure 4. Cost gap to Claude Sonnet 5.5, by factor. Dollars per success = trials per pass × steps per trial × tokens per step × price per token after caching. Log scale.
After caching, price per token is flat. Steps and context set the cost.
Against Sonnet: Flash 11.7× per success (8.7× the steps), Argon 9.2× (3.1× the steps, 2.8× the context), Muse 6.5× (3.7× the steps). With 83% or more of input cached, a step costs what its context costs.
Median25th to 75th percentilePrompt tokens per call against position in the trajectory, first turn to last
Figure 5. Context per step over a trajectory. Largest prompt sent so far; median, middle half shaded.
Argon keeps working after its fix passes
123 of Argon's 600 trials hit the 2-hour limit; 90 of those passed. The fix was in, and Argon kept re-checking with context near 400k tokens a call. Knowing when it is done is worth more to Argon than a better fix.
Passing trialsFailing trialsRatio, failing trials over passing trials
Figure 6. Steps when passing and when failing. Median, log scale.
Failing trials run as long as passing ones
Every pair takes 1.1 to 1.2× the steps when it fails.
Reading and searchingEditingRunning testsEnvironment, services, waitsOther
Figure 7. Tool calls by phase. Classified from each call's command and target; a call that reads and writes counts as an edit.
Sonnet edits halfway in and stops soon after
On a passing trial Sonnet first edits halfway through and stops 5 steps after its last edit. Argon stops 13 steps after, Muse 24, Flash 46.
A third of Flash's replies never run
Terminus 2 cannot parse 33% of Flash's replies into a command; each costs a step and does nothing. Most of Flash's cost gap is steps.
| Tier · concern | Concurrency and duplicates | Failures and deadlines | Stored data and rollout | Access control | Business logic only | All |
|---|---|---|---|---|---|---|
| Easyunder 2 hours | 0.56×11 tasks | 0.40×14 tasks | 0.78×9 tasks | 0.42×5 tasks | 0.51×11 tasks | 0.52×50 tasks |
| Medium2 to 3½ hours | 0.91×44 tasks | 1.46×5 tasks | 1.16×4 tasks | 0.79×4 tasks | 1.27×2 tasks | 0.96×59 tasks |
| Hard3½ hours or more | 1.95×36 tasks | 2.75×4 tasks | 3.53×1 task | – | – | 2.05×41 tasks |
| All | 1.18×91 tasks | 0.80×23 tasks | 0.99×14 tasks | 0.55×9 tasks | 0.59×13 tasks | 1.00×150 tasks |
Table 3. Dollars per solve by difficulty and engineering concern. Over each pair's own average solve, then averaged over pairs.
03Environment
The tasks are built for scaling RL, built on TrueReward, our environment infrastructure, that includes high fidelity enterprise system runtimes. We present below the runtimes used in EnterpriseSWE.
Task
The task statement
Agent container
- The service tree, its migrations and client code
- The stack up locally, to build and run against
- A shell through the harness under test
Grader container
- Rebuild the changed services
- Run copies; kill, freeze, cut, stall and shift the clock as the case says
- Run the must-fix and must-not-break cases
Reward
1 if every case passes, else 0
Runtimes
- PostgreSQL 17, seeded
- Redis 8
- Kafka
- DynamoDB Local, S3, SQS, SES
- Payment and payout providers
- Process copies, faults, clock
- Sandbox
- One for the agent and one for the grader, per trial. 0.5 core and 3.4 GiB guaranteed; 2 to 4 cores and 4 to 8 GiB cap, per task. No network.
- Budget
- Two hours of agent time per trial. At the limit the agent is stopped and the grader scores the tree it left.
- Grading
- Rebuild, then every hidden case. Median 2½ minutes, 95th percentile 15.
- Reward
- Binary. r = 1 when every fail-to-pass case and every pass-to-pass case passes, else 0. Fail-to-pass cases test the requested change; pass-to-pass cases pin behaviour that must not break.
- Rollout cost
- $0.03 to $0.24 of sandbox compute a rollout, 2 to 4% of model spend; not in the leaderboard.
- Task CI
- Candidates mined from the codebase. Tests written from the task statement. A separate solver writes the reference blind, from the statement and starting code only. Tests and solutions play against each other until neither finds a new disagreement. Task CI scores every arm. Authored with Kennet.
- Agent
- Claude Sonnet 5.5 on Claude Code 2.1.288, medium effort; Gemini 4 Argon on Antigravity CLI 1.3.5, high effort; Muse Spark 1.3 on Muse Code 1.4.7, medium effort; Gemini 3.8 Flash on Terminus 2 2.0.0, medium effort. 4 trials per task.
- Score
- Share of passing trials, averaged over tasks.
- Interval
- 95% bootstrap over tasks. Shared rank unless intervals separate.
- Infra failures
- 1 of 2,400 trials, left out of every score.
How agents try to cheat
We studied a few trajectories and present an early preview of our research on reward hacking.
- Special-case the fixtures
- Branch on the IDs the local data seeds. Hidden cases bring their own data.
- Change what the tests observe
- A process-wide override the tests share, such as swapping the HTTP client. Refused before grading.
04Task construction
- Hidden cases
- 2,938 (2,026 fail-to-pass, 912 pass-to-pass). Median 19 per task.
- Known-wrong
- 3,200 plausible wrong fixes, each a realistic mistake, that the verifier must reject. Median 21 per task. Every one is rejected.
- Known-right
- 366 correct fixes besides the reference, written differently, that the verifier must accept; 228 written blind from the statement alone. Every one is accepted.
Every task sits on two axes read from its own files, never from model results, so pairs and releases share one scale.
Concern: the hardest systems condition the statement states (concurrency, failures and deadlines, stored data and rollout, access control), else business logic only. Difficulty: an experienced engineer's time to design and build the fix, the geometric mean of four blind estimates. Easy under 2 hours, medium 2 to 3½, hard over 3½.
05Future work
- We plan to partner with more enterprises to broaden both the depth and breadth of our private codebase work. If you are a technical leader interested in researching frontier agents on production incidents, please reach out to us here.
- We plan to publish a deeper analysis on our research on reward hacking attempts by the solver agents in our work.
- We will expand our bench to close frontier open source models and extend our failure analysis with contrastive analysis on successful attempts and failure modes across agents.
06References
- Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. AI Agents That Matter. arXiv:2407.01502, 2024.
- Kapoor, S., Stroebl, B., et al. Holistic Agent Leaderboard. Princeton University.
Cite
@misc{metaphi2026enterpriseswe,
title = {{EnterpriseSWE} v0.3.0},
author = {{Metaphi AI}},
year = {2026},
month = oct,
howpublished = {\url{https://metaphi.ai/benchmarks/enterpriseswe/backend}},
note = {Version v0.3.0}
}