Takeaways
- Frontier agents fail 54% of trials. 89% of those failures sit outside the requested change: the old build still serving, a slow or restarting service, overlapping work, or code next to the change. In 85% the agent never ran the condition that failed.
- GPT-6.1 Sol on Codex costs least per success, $2.50, a third of Claude Opus 5.5's $7.62. Opus is the most accurate, passing 59% of trials.
- Agents are not yet consistent. Opus solves 71% of tasks in at least one of four tries, and 43% in all four.
01Effectiveness
| Tasks | Claude Opus 5.5Claude Code | GPT-6.1 SolCodex | Muse Spark 1.3Muse Code | Gemini 3.8 FlashTerminus 2 |
|---|---|---|---|---|
| All tasks | ||||
| Success rate, pass@1 | 59% | 34% | 29% | 20% |
| Ceiling, pass@4 | 71% | 46% | 46% | 39% |
| Reliability, pass^4 | 43% | 21% | 11% | 4% |
| Difficulty | ||||
| Easy2 tasks | 100% | 100% | 50% | 25% |
| Medium7 tasks | 61% | 39% | 25% | 25% |
| Hard19 tasks | 54% | 25% | 28% | 17% |
| Systems conditions the statement demands | ||||
| Concurrent requests16 tasks | 56% | 30% | 34% | 25% |
| A process killed mid-work12 tasks | 48% | 15% | 38% | 29% |
| Several copies of a service11 tasks | 50% | 18% | 23% | 23% |
| A slow or failing dependency14 tasks | 50% | 25% | 21% | 7% |
| A stated deadline19 tasks | 55% | 33% | 26% | 20% |
| None of the five3 tasks | 100% | 92% | 42% | 17% |
| Three or more14 tasks | 48% | 27% | 34% | 23% |
| Avg rank over tasks1 is best | 1.82 | 2.57 | 2.70 | 2.91 |
0 of 41 of 42 of 43 of 44 of 4
Figure 1. Tasks by trials passed. Each task by how many of the pair's 4 trials passed.
- Claude Opus 5.5Claude Code112
- GPT-6.1 SolCodex112
- Gemini 3.8 FlashTerminus 2108
| Guards held, share of must-fix cases fixed | under 25% | 25 to 50% | 50 to 75% | 75% or more |
|---|---|---|---|---|
| Claude Opus 5.5 | 0 | 0 | 4 | 13 |
| GPT-6.1 Sol | 4 | 1 | 6 | 12 |
| Muse Spark 1.3 | 3 | 0 | 18 | 34 |
| Gemini 3.8 Flash | 3 | 2 | 8 | 28 |
| Where failures fall | Claude Opus 5.5Claude Code | GPT-6.1 SolCodex | Muse Spark 1.3Muse Code | Gemini 3.8 FlashTerminus 2 |
|---|---|---|---|---|
| Failing trials | 46 | 74 | 80 | 90 |
| On tasks the pair never passes | 32 | 60 | 60 | 68 |
| On tasks the pair passes in another trial | 14 | 14 | 20 | 22 |
| Failure mode | Claude Opus 5.5Claude Code | GPT-6.1 SolCodex |
|---|---|---|
| Backward compatibilityFails against an earlier version still in use: the old build serving, or the data it wrote. | 17 | 16 |
| Workload, recovery and restartFails near a resource limit, after a fault, or on restart. | 14 | 19 |
| Side effectsBreaks behaviour outside the changed function, in ordinary use. | 7 | 26 |
| ConcurrencyFails where work on a shared resource overlaps: concurrent requests, copies, locks. | 5 | 3 |
| Design conformanceDeparts from the stated requirement. | 3 | 5 |
| Nothing shippedEnds with no change to test. | 0 | 5 |
| Failure mode | Reading the task | Design | Implementation | Verification | Process | All |
|---|---|---|---|---|---|---|
| Backward compatibility | 16 | 14 | – | 3 | – | 33 |
| Workload, recovery and restart | 7 | 23 | 3 | – | – | 33 |
| Side effects | 14 | 14 | 5 | – | – | 33 |
| Concurrency | – | 7 | 1 | – | – | 8 |
| Design conformance | 6 | – | 2 | – | – | 8 |
| Nothing shipped | – | – | – | – | 5 | 5 |
| All | 43 | 58 | 11 | 3 | 5 | 120 |
Table 3. Where the decisive mistake sits. Failed trials by failure mode; each trial counted once.
| Failure mode | Broke existing behaviour | Fix incomplete | Did not build or start | Nothing shipped | All |
|---|---|---|---|---|---|
| Backward compatibility | 17 | 13 | 3 | – | 33 |
| Workload, recovery and restart | 24 | 9 | – | – | 33 |
| Side effects | 29 | 4 | – | – | 33 |
| Concurrency | 1 | 7 | – | – | 8 |
| Design conformance | 1 | 7 | – | – | 8 |
| Nothing shipped | – | – | – | 5 | 5 |
| All | 72 | 40 | 3 | 5 | 120 |
Table 4. What the verifier saw. Failed trials by failure mode; each trial counted once.
| Failure mode | Done, no caveat | Flagged a doubt | Did not claim done | All |
|---|---|---|---|---|
| Backward compatibility | 14 | 19 | – | 33 |
| Workload, recovery and restart | 19 | 14 | – | 33 |
| Side effects | 25 | 8 | – | 33 |
| Concurrency | 3 | 5 | – | 8 |
| Design conformance | 6 | 2 | – | 8 |
| Nothing shipped | – | – | 5 | 5 |
| All | 67 | 48 | 5 | 120 |
Table 5. What the agent said at the end. Failed trials by failure mode; each trial counted once.
| Failure mode | Ran it | Never ran it | All |
|---|---|---|---|
| Backward compatibility | 7 | 26 | 33 |
| Workload, recovery and restart | 2 | 31 | 33 |
| Side effects | 3 | 30 | 33 |
| Concurrency | 1 | 7 | 8 |
| Design conformance | 2 | 6 | 8 |
| Nothing shipped | 3 | 2 | 5 |
| All | 18 | 102 | 120 |
Table 6. Whether the agent ran the condition that failed. Failed trials by failure mode; each trial counted once.
In the agent's words
Modes are the triggers of Orthogonal Defect Classification 1, the condition under which a defect surfaces; workload/stress, recovery/exception and startup/restart are merged. The verifier's reading follows the fault, error, failure chain 2 and stages follow Liu et al. 3. Agent-failure taxonomies, built on single-version repositories, rarely see these triggers 4; in human-written systems they are common and severe 5 6, and distributed-systems repair shows the same gap between source and runtime 7.
A worker stopped between the credit and those steps lost them for good, because a resent notice found the deposit already finished.
The core path works: 503 at 4.0s with Retry-After: 1, no app sessions left 1s after the answer, and nothing written.Fixture setup also ignores squad and injury fetch errors ... I've made those failures stop setup before saving
02Efficiency
Pareto frontierCheaper and more accurate than the median pair95% interval
Figure 3. Success rate against dollars per success. Ringed pairs are the Pareto frontier.
Trials per passSteps per trialTokens per stepPrice per token, after cachingTotal, the product, in the pair’s hue
Figure 4. Cost gap to GPT-6.1 Sol, by factor. Dollars per success = trials per pass × steps per trial × tokens per step × price per token after caching. Log scale.
Figure 5. Context per step over a trajectory. Largest prompt sent so far; median, middle half shaded.
Passing trialsFailing trialsRatio, failing trials over passing trials
Figure 6. Steps when passing and when failing. Median, log scale.
Reading and searchingEditingRunning testsEnvironment, services, waitsOther
Figure 7. Tool calls by phase. Classified from each call's command and target; a call that reads and writes counts as an edit.
| Tier · concern | Concurrency and duplicates | Failures and deadlines | Stored data and rollout | Business logic only | All |
|---|---|---|---|---|---|
| Easyunder 2 hours | – | – | 0.24×1 task | 0.21×1 task | 0.37×2 tasks |
| Medium2 to 3½ hours | 0.75×5 tasks | 0.66×2 tasks | – | – | 0.70×7 tasks |
| Hard3½ hours or more | 1.20×15 tasks | 2.40×3 tasks | 0.62×1 task | – | 1.29×19 tasks |
| All | 1.08×20 tasks | 1.30×5 tasks | 0.52×2 tasks | 0.21×1 task | 1.00×28 tasks |
Table 7. Dollars per solve by difficulty and engineering concern. Over each pair's own average solve, then averaged over pairs.
03Task construction
- Hidden cases
- 561 (376 fail-to-pass, 185 pass-to-pass). Median 19 per task.
- Known-wrong
- 650 plausible wrong fixes, each a realistic mistake, that the verifier must reject. Median 21 per task. Every one is rejected.
- Known-right
- 65 correct fixes besides the reference, written differently, that the verifier must accept; 0 written blind from the statement alone. Every one is accepted.
Every task sits on two axes read from its own files, never from model results, so pairs and releases share one scale.
Concern: the hardest systems condition the statement states (concurrency, failures and deadlines, stored data and rollout, access control), else business logic only. Difficulty: an experienced engineer's time to design and build the fix, the geometric mean of four blind estimates. Easy under 2 hours, medium 2 to 3½, hard over 3½.
One task as the agent receives them: the task's id, its statement or a passage of it, and how many of each pair's trials passed.
**First-deposit referral rewards and deposit-offer bonuses go missing**
Some referrers are never paid the first-deposit referral reward after a friend who joined with their code deposits, and some players who chose a deposit offer for their first deposit get no bonus. It happens when a player has more than one deposit open at once, or when a deposit is processed while the wallet workers restart.
Expected: a player's deposits are numbered in the order they are credited to the wallet, not the order they were started. When a referred player's first credited deposit lands, their referrer gets the first-deposit reward once, on today's terms, and no other deposit pays it. […] A paid deposit, with its bonus and any referral reward, is in the wallet within a minute of the provider's paid notice; a deposit whose handling was cut off is finished when the provider resends that notice. […]
The fix must be correct when several copies of the wallet worker run at the same time, and when any copy is stopped at any point. Schema changes must work while today's build is still running.
04Environment
The tasks are built for scaling RL, built on TrueReward, our environment infrastructure, that includes high fidelity enterprise system runtimes. We present below the runtimes used in EnterpriseSWE.
Task
A business request, as the business states it
Agent container
- The service tree, its migrations and client code
- The stack up locally, to build and run against
- A shell through the harness under test
Grader container
- Rebuild the changed services
- Run copies; kill, freeze, cut, stall and shift the clock as the case says
- Run the must-fix and must-not-break cases
Reward
1 if every case passes, else 0
Runtimes
- PostgreSQL 17, seeded
- Redis 8
- Kafka
- DynamoDB Local, S3, SQS, SES
- Payment and payout providers
- Process copies, faults, clock
- Sandbox
- One for the agent and one for the grader, per trial. 0.5 core and 3.4 GiB guaranteed; 2 to 4 cores and 4 to 8 GiB cap, per task. No network.
- Budget
- Two hours of agent time per trial. At the limit the agent is stopped and the grader scores the tree it left.
- Grading
- Rebuild, then every hidden case. Median 2½ minutes, 95th percentile 15.
- Reward
- Binary. r = 1 when every fail-to-pass case and every pass-to-pass case passes, else 0. Fail-to-pass cases test the requested change; pass-to-pass cases pin behaviour that must not break.
- Rollout cost
- $0.03 to $0.24 of sandbox compute a rollout, 2 to 4% of model spend; not in the leaderboard.
- Task CI
- Candidates mined from the codebase. Tests written from the task statement. A separate solver writes the reference blind, from the statement and starting code only. Tests and solutions play against each other until neither finds a new disagreement. Task CI scores every arm. Authored with Kennet.
- Agent
- Claude Opus 5.5 on Claude Code 2.1.310; GPT-6.1 Sol on Codex 0.164.0; Muse Spark 1.3 on Muse Code 1.4.7; Gemini 3.8 Flash on Terminus 2 2.0.0. Medium reasoning effort for every pair. 4 trials per task.
- Score
- Share of passing trials, averaged over tasks.
- Interval
- 95% bootstrap over tasks. Shared rank unless intervals separate.
- Infra failures
- 0 of 448 trials, left out of every score.
05References
- Chillarege, R., Bhandari, I. S., Chaar, J. K., Halliday, M. J., Moebus, D. S., Ray, B. K., and Wong, M.-Y. Orthogonal Defect Classification: A Concept for In-Process Measurements. IEEE Transactions on Software Engineering 18(11), 1992.
- Avizienis, A., Laprie, J.-C., Randell, B., and Landwehr, C. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Transactions on Dependable and Secure Computing 1(1), 2004.
- Liu, S., Liu, F., Li, L., Tan, X., Zhu, Y., Lian, X., and Zhang, L. An Empirical Study on Failures in Automated Issue Solving. arXiv:2509.13941, 2025.
- Cemri, M., Pan, M. Z., Yang, S., et al. Why Do Multi-Agent LLM Systems Fail? NeurIPS Datasets and Benchmarks, 2025.
- Zhang, Y., Yang, J., Jin, Z., Sethi, U., Rodrigues, K., Lu, S., and Yuan, D. Understanding and Detecting Software Upgrade Failures in Distributed Systems. SOSP, 2021.
- Leesatapornwongsa, T., Lukman, J. F., Lu, S., and Gunawi, H. S. TaxDC: A Taxonomy of Non-Deterministic Concurrency Bugs in Datacenter Distributed Systems. ASPLOS, 2016.
- Yan, Y., Wang, H., He, J., Liang, Y., Wang, S., Sun, H., and Park, S. J. Evaluating Agentic Code Repair Capabilities in Distributed Systems. arXiv:2608.14863, 2026.
- Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. AI Agents That Matter. arXiv:2407.01502, 2024.
- Kapoor, S., Stroebl, B., et al. Holistic Agent Leaderboard. Princeton University.
Cite
@misc{metaphi2026enterpriseswe,
title = {{EnterpriseSWE} v0.7.0},
author = {{Metaphi AI}},
year = {2026},
month = oct,
howpublished = {\url{https://metaphi.ai/benchmarks/enterpriseswe/backend}},
note = {Version v0.7.0}
}