EnterpriseSWE
How effective, and efficient are agents on daily engineering PRs in private production codebases?
- 1Claude Sonnet 5.5Claude Code · medium effort85.3%150 tasks × 4 trials · $0.77 per trial · 2026-10-16
- 1Gemini 4 ArgonAntigravity CLI · high effort79.5%150 tasks × 4 trials · $6.62 per trial · 2026-10-16
- 3Gemini 3.8 FlashTerminus 2 · medium effort59.5%150 tasks × 3 to 4 trials · $6.32 per trial · 2026-10-16
Takeaways
- A success costs $0.91 with Sonnet on Claude Code, $5.87 with Muse Spark 1.3 on Muse Code, $8.33 with Argon and $10.60 with Gemini 3.8 Flash on Terminus 2.
- List prices differ by up to 2.7×, but once caching is counted every pair pays within 20% of the others per token. Steps per trial (30 to 260) and tokens per step (103k to 287k) make up the gap.
- Failing trials run as long as passing ones, or longer. No pair stops sooner when it is failing.
- Argon keeps going after its fix passes: 123 of its 600 trials ran into the 2-hour limit, and 90 of those passed.
01Efficiency
| Per pair | Claude Sonnet 5.5Claude Code | Gemini 4 ArgonAntigravity CLI | Muse Spark 1.3Muse Code | Gemini 3.8 FlashTerminus 2 |
|---|---|---|---|---|
| Outcome | ||||
| Success rate, pass@1higher is better | 85%81%–90% | 80%74%–85% | 63%57%–69% | 60%53%–66% |
| Dollars per successlower is better | $0.91$0.74–$1.12 | $8.33$7.39–$9.38 | $5.87$5.07–$6.86 | $10.60$9.18–$12.38 |
| Dollars per triallower is better | $0.77 | $6.62 | $3.70 | $6.32 |
| Where the spend goes | ||||
| Steps per triallower is better | 30 | 92 | 112 | 260 |
| Steps to a pass, medianlower is better | 23 | 84 | 95 | 246 |
| Tokens per steplower is better | 103k | 287k | 158k | 118k |
| Tokens per triallower is better | 3.1M | 26M | 18M | 31M |
| Input read from cachehigher is better | 97% | 95% | 97% | 83% |
| Time | ||||
| Active minutes per trial, medianlower is better | 3.7 min | 53 min* | 12 min† | 51 min |
| Trials stopped at the 2-hour limitlower is better | 0 | 12390 passed | 0 | 31 passed |
* Gemini 4 Argon also waited a median 9.7 min per trial on provider capacity.† Muse Spark 1.3 also waited a median 21 min per trial on provider capacity.
Table 1. Efficiency by pair. Ranges are 95% bootstrap intervals over tasks.
Trials per passSteps per trialTokens per stepPrice per token, after cachingTotal, the product, in the pair’s hue
Figure 1. Cost gap to Claude Sonnet 5.5, by factor. Dollars per success = trials per pass × steps per trial × tokens per step × price per token after caching. Log scale.
Cached inputFresh inputCache writesOutput and reasoning
Figure 2. Tokens per trial, by kind. Mean over scored trials.
Median25th to 75th percentilePrompt tokens per call against position in the trajectory, first turn to last
Figure 3. Context per step over a trajectory. Largest prompt sent so far; median, middle half shaded.
Reading and searchingEditingRunning testsEnvironment, services, waitsOther
Figure 4. Tool calls by phase. Classified from each call's command and target.
Passing trialsFailing trialsRatio, failing trials over passing trials
Figure 5. Steps when passing and when failing. Median, log scale.
Passing trial, upper halfFailing trial, lower halfMedian, with n at the right
Figure 6. Cost of every trial. Log scale.
Pareto frontierCheaper and more accurate than the median pair95% interval
Figure 7. Success rate against dollars per success. Ringed pairs are the Pareto frontier.
Pareto frontierFaster and more accurate than the median pair95% interval
Figure 8. Success rate against active minutes per trial. Model and tool time, median.
Waiting is not working
Median minutes waiting for the provider to accept a call: Muse 21, Argon 10, Sonnet and Flash 0. Muse's median trial takes 32 minutes end to end, of which 12 are model and tool time. Argon averages 19 provider refusals per trial, each retried by the harness.
Rollout throughput for RL depends on these waits, but they are not the agent's behaviour, so the time scatter plots active minutes.
02Environment
Enterprise backend work happens in context: one business rule spreads across services, jobs, migrations and configuration, and must hold under concurrent requests, a process killed mid-write, several copies of a service, a slow database, a deadline. Each task is such a change, worded as the business asks for it, and is graded on the running system under those conditions.
- System
- Production backend services with their datastores; third-party providers are stand-ins the grader controls. This release: five systems from two businesses, 9k to 110k lines of Go over PostgreSQL, Redis and DynamoDB, and a Flutter client.
- Observation
- The service tree, the task statement, and a local stack to build and run the services against. The statement states the business outcome and the conditions it must hold under, never the fix.
- Action
- A shell in the agent's container, through the harness under test.
- Hidden
- The cases, written from the statement, and the reference fix.
- Budget
- Two hours of wall clock per trial. No network. A trial that runs out of time scores 0.
- Reward
- Binary. r = 1 when every fail-to-pass case and every pass-to-pass case passes, else 0. Fail-to-pass cases test the requested change; pass-to-pass cases pin behaviour that must not break.
- Grader
- A separate container rebuilds the changed services, brings the stack up, and runs the cases under the conditions the statement names: several copies of a binary, each with its own user and port, so a local lock cannot fake a fix; requests released together; a copy killed or frozen mid-write; a lock held or a session killed; datastores and providers cut, stalled, throttled or answering late; a write failed before or after it applies. Deadlines are asserted on the wall clock.
Task
The task statement
Agent container
- Service source, migrations and client code
- A local stack to build and run against
- A shell through the harness under test
Grader container
- Rebuild the changed services
- Bring the stack up; kill, freeze, replicate, delay as the statement names
- Run the must-fix and must-not-break cases
Reward
1 if every case passes, else 0
- Task CI
- Candidates mined from the codebase. Tests written from the task statement. A separate solver writes the reference blind, from the statement and starting code only. Tests and solutions play against each other until neither finds a new disagreement. Task CI scores every arm. Authored with Kennet.
- Environment
- Production backend services with their tasks and tests, run in containers. No network.
- Agent
- Claude Sonnet 5.5 on Claude Code 2.1.288, medium effort; Gemini 4 Argon on Antigravity CLI 1.3.5, high effort; Muse Spark 1.3 on Muse Code 1.4.7, medium effort; Gemini 3.8 Flash on Terminus 2 2.0.0, medium effort. 4 trials per task.
- Score
- Share of passing trials, averaged over tasks.
- Interval
- 95% bootstrap over tasks. Shared rank unless intervals separate.
03Task construction
- Hidden cases
- 2,938 (2,026 fail-to-pass, 912 pass-to-pass). Median 19 per task.
- Known-wrong
- 3,200 plausible wrong fixes, each a realistic mistake, that the verifier must reject. Median 21 per task. Every one is rejected.
- Known-right
- 366 correct fixes besides the reference, written differently, that the verifier must accept; 228 written blind from the statement alone. Every one is accepted.
Every task sits on two axes read from its own files, never from model results, so pairs and releases share one scale.
Engineering concern: the hardest systems condition the statement states, taken in this order: concurrent writers, duplicate requests or several copies of a service; a crash, a failing dependency or a deadline; data already stored or old and new builds side by side; who may see or do what. A task that states none is business logic only. Difficulty: the time an experienced engineer needs to design and implement the fix, the geometric mean of four blind estimates from two model families. Easy is under 2 hours, medium 2 to 3½, hard 3½ or more; cutoffs are fixed across releases.
| Tier · concern | Concurrency and duplicates | Failures and deadlines | Stored data and rollout | Access control | Business logic only | All |
|---|---|---|---|---|---|---|
| Easyunder 2 hours | 11 | 14 | 9 | 5 | 11 | 50 |
| Medium2 to 3½ hours | 44 | 5 | 4 | 4 | 2 | 59 |
| Hard3½ hours or more | 36 | 4 | 1 | – | – | 41 |
| All | 91 | 23 | 14 | 9 | 13 | 150 |
Table 2. Tasks by difficulty and engineering concern. A task's concern is the hardest systems condition its statement states.
04Results by axis
Success rate averaged over each row's tasks; best pair in bold. Average rank ranks the pairs on each task by pass rate, ties shared. A task counts under every condition it demands.
| Tier · concern | Concurrency and duplicates | Failures and deadlines | Stored data and rollout | Access control | Business logic only | All |
|---|---|---|---|---|---|---|
| Easyunder 2 hours | 0.8111 tasks | 0.9214 tasks | 0.809 tasks | 0.935 tasks | 0.8311 tasks | 0.8550 tasks |
| Medium2 to 3½ hours | 0.7644 tasks | 0.595 tasks | 0.724 tasks | 0.724 tasks | 0.532 tasks | 0.7359 tasks |
| Hard3½ hours or more | 0.5536 tasks | 0.454 tasks | 0.381 task | – | – | 0.5341 tasks |
| All | 0.6891 tasks | 0.7723 tasks | 0.7514 tasks | 0.839 tasks | 0.7813 tasks | 0.72150 tasks |
Table 3. Average reward by difficulty and engineering concern. Every trial of the 4 pairs on the cell's tasks.
| Tier · concern | Concurrency and duplicates | Failures and deadlines | Stored data and rollout | Access control | Business logic only | All |
|---|---|---|---|---|---|---|
| Easyunder 2 hours | 0.56×11 tasks | 0.40×14 tasks | 0.78×9 tasks | 0.42×5 tasks | 0.51×11 tasks | 0.52×50 tasks |
| Medium2 to 3½ hours | 0.91×44 tasks | 1.46×5 tasks | 1.16×4 tasks | 0.79×4 tasks | 1.27×2 tasks | 0.96×59 tasks |
| Hard3½ hours or more | 1.95×36 tasks | 2.75×4 tasks | 3.53×1 task | – | – | 2.05×41 tasks |
| All | 1.18×91 tasks | 0.80×23 tasks | 0.99×14 tasks | 0.55×9 tasks | 0.59×13 tasks | 1.00×150 tasks |
Table 4. Dollars per solve by difficulty and engineering concern. Each pair's dollars per solve on the cell's tasks over its own on every task, averaged over the pairs; 1× is a pair's average solve.
| Tasks | Claude Sonnet 5.5Claude Code | Gemini 4 ArgonAntigravity CLI | Muse Spark 1.3Muse Code | Gemini 3.8 FlashTerminus 2 |
|---|---|---|---|---|
| All tasks | ||||
| Success rate, pass@1 | 85% | 80% | 63% | 60% |
| Ceiling, pass@3 | 93% | 89% | 80% | 71% |
| Reliability, pass^3 | 74% | 69% | 45% | 48% |
| Difficulty | ||||
| Easy50 tasks | 91% | 83% | 84% | 84% |
| Medium59 tasks | 86% | 86% | 63% | 57% |
| Hard41 tasks | 77% | 65% | 38% | 33% |
| Systems conditions the statement demands | ||||
| Concurrent requests72 tasks | 83% | 75% | 57% | 51% |
| A process killed mid-work34 tasks | 82% | 74% | 58% | 46% |
| Several copies of a service44 tasks | 86% | 78% | 54% | 50% |
| A slow or failing dependency64 tasks | 86% | 75% | 56% | 53% |
| A stated deadline60 tasks | 81% | 76% | 54% | 54% |
| None of the five39 tasks | 90% | 83% | 74% | 67% |
| Three or more50 tasks | 86% | 75% | 54% | 47% |
| Avg rank over tasks1 is best | 2.12 | 2.22 | 2.79 | 2.86 |
05Failure analysis
Four model × harness pairs on the same 150 tasks, four trials each. Claude Sonnet 5.5 on Claude Code passes 85% of trials and Gemini 4 Argon on Antigravity CLI 80%, but Argon spends nine times as much per success. Every pair pays about the same per token. The gap in what a success costs comes from how many steps a pair takes and how much context it sends on each one.
- Claude Sonnet 5.5Claude Code528
- Gemini 4 ArgonAntigravity CLI537
- Gemini 3.8 FlashTerminus 2527
| Guards held, share of must-fix cases fixed | under 25% | 25 to 50% | 50 to 75% | 75% or more |
|---|---|---|---|---|
| Claude Sonnet 5.5 | 1 | 2 | 7 | 49 |
| Gemini 4 Argon | 0 | 0 | 4 | 34 |
| Muse Spark 1.3 | 11 | 4 | 29 | 111 |
| Gemini 3.8 Flash | 7 | 2 | 18 | 109 |
| Tool calls | Claude Sonnet 5.5Claude Code | Gemini 4 ArgonAntigravity CLI | Muse Spark 1.3Muse Code | Gemini 3.8 FlashTerminus 2 |
|---|---|---|---|---|
| Calls per trial, median | 24 | 140 | 118 | 201 |
| Calls that failed | 11.2% | 5.0% | 6.3% | 5.8% |
| Failed calls followed by a working call on the same target within five calls | 0% | 0% | 0% | 0% |
| Replies the harness could not parse into a call, per trial, median | 0 | 0 | 0 | 88 |
After caching, price per token is flat. Steps and context set the cost.
Dollars per success is the product of four factors: trials per pass, steps per trial, tokens per step, and dollars per token after caching (spend over all tokens, cached reads included). Measured against Sonnet:
- Flash costs 11.7× as much per success: 8.7× the steps, 1.4× the trials per pass, 1.1× the tokens per step, 0.8× the price per token.
- Argon costs 9.2×: 3.1× the steps and 2.8× the tokens per step. Its price per token and its pass rate are within 7% of Sonnet's.
- Muse costs 6.5×: 3.7× the steps, 1.5× the tokens per step, 1.4× the trials per pass.
Every pair reads more than four in five input tokens from cache, so the cost of a step is mostly the context it carries. A model that commits sooner and carries less context is cheaper on every factor that counts. Those two factors are what training can move.
Failing trials run as long as passing ones
Median steps, passing against failing: Sonnet 23 and 27, Argon 84 and 101, Muse 95 and 113, Flash 246 and 266. A failing trial costs at least what a pass costs and returns a reward of 0.
For a policy in training, every failing rollout is spent compute with no positive signal. A model that recognises it is stuck and stops sooner wastes less of the sampling budget.
Argon keeps working after its fix passes
Argon's median trial runs 53 active minutes; Sonnet's runs 4. 123 of Argon's 600 trials ran into the 2-hour limit, and 90 of those passed: the fix was in place, and Argon kept reading and re-checking until the clock stopped it.
Its context grows the most of the four, to about 400k tokens a call by the end of a trajectory. Each extra step after a working fix costs more than the step before it. Learning when it is done is worth more to Argon than learning a better fix.
Sonnet edits halfway in and stops soon after
On a passing trial, Sonnet makes its first edit to product code halfway through the trajectory and finishes 5 steps after its last edit. Argon edits at 69% and finishes 13 steps after its last edit; Flash at 62% and 46 steps after. Muse edits earliest, at 37%, then takes 24 more steps.
Edits are a quarter of Sonnet's and Muse's tool calls, 13% of Argon's and 9% of Flash's. Most of every pair's calls read and search the code.
A third of Flash's replies never run
Terminus 2 cannot parse 33% of Flash's replies into a command, so the step costs tokens and does nothing. Flash also spends 18% of its calls on the environment and its services, nearly half of them waiting on a process to finish.
Flash takes 260 steps a trial. Its tokens per step and price per token are close to Sonnet's. Most of its cost gap is steps, and a third of those steps are rejected replies. A reward on the harness's reply format targets these steps directly.
06Limits
- Each row is a model on its own harness. A gap belongs to the pair, not to the model alone.
- Argon runs at high reasoning effort, the only setting Antigravity CLI offers.
- Spend is each trial's tokens at the provider's list price on the run date. Argon's and Flash's prices on those dates were introductory; at their later list prices both cost twice as much per token.
- For one Flash run the gateway recorded fewer tokens than the trajectory, so those trials are priced from the trajectory's counts, with reasoning folded into output.
- Muse Code ran at versions 1.4.7 and 1.4.9. It makes about two model calls per trajectory step; a step counts once.
- Phases come from classifying each tool call by its command and target. A call that both reads and writes counts as an edit.
07Future work
- Every model on one harness (Terminus 2), to separate the model's share of a gap from the harness's.
- Effort sweeps where a harness offers them, plotted as accuracy against output tokens.
- A smaller open model on the same tasks, the size a post-training team starts from.
08References
- Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. AI Agents That Matter. arXiv:2407.01502, 2024.
- Kapoor, S., Stroebl, B., et al. Holistic Agent Leaderboard. Princeton University.
Cite
@misc{metaphi2026enterpriseswe,
title = {{EnterpriseSWE} v0.3.0},
author = {{Metaphi AI}},
year = {2026},
month = oct,
howpublished = {\url{https://metaphi.ai/benchmarks/enterpriseswe/backend}},
note = {Version v0.3.0}
}