Metaphi AI
← Benchmarks
Benchmark · Business problem solving

CompanyGym

How well do frontier agents fare on day-to-day business problems?

Evaluate your agent

OutcomeEfficiency
#Model · harness
1
Claude Sonnet 5.5Claude Code · medium effort
9.1%2–18%15.2%5–28%3.0%0–8%$12.30$5.22–$55.67$1.123.5M18
1
Qwen 3.8 27BClaude Code · medium effort
1.1%0–3%2.3%0–7%0.0%0–0%$21.01$5.84 or more$0.251.8M17

Table 1. Leaderboard. Ranges are 95% bootstrap intervals over tasks; # ranks on success, pairs whose intervals overlap sharing a rank. Any column sorts.

Source: CompanyGym v0.2.0, 172 trials · metaphi.ai
Contents
  1. Leaderboard
  2. Effectiveness
  3. Efficiency
  4. Task construction
  5. Environment design
  6. Reward design
  7. Limits
  8. Future work
  9. References

Takeaways

  1. The first failure is no deliverable. Qwen 3.8 27B wrote no memo in 70% of its trials: it gathers evidence too slowly to reach the point where it writes.
  2. Given the evidence, agents reason well. Wrong inference causes 4 of the best configuration's 80 failures.
  3. They fail on what they never look for. In 21 of 22 root-cause misses we traced, the deciding record was never opened.
  4. They size the right driver with the nearest figure, and stop at a diagnosis where the user needs a decision.

01Effectiveness

TasksClaude Sonnet 5.5Claude CodeQwen 3.8 27BClaude Code
All tasks
Success rate, pass@19%1%
Ceiling, pass@215%2%
Reliability, pass^23%0%
Avg rank over tasks1 is best1.391.61
0510152025TasksClaude Sonnet 5.5Claude CodeClaude Sonnet 5.5, 0 of 4: 17 (77%)17Claude Sonnet 5.5, 1 of 4: 3 (14%)3Claude Sonnet 5.5, 2 of 4: 1 (5%)Claude Sonnet 5.5, 3 of 4: 1 (5%)5 mixedQwen 3.8 27BClaude CodeQwen 3.8 27B, 0 of 4: 18 (95%)18Qwen 3.8 27B, 1 of 4: 1 (5%)1 mixed
0102030TasksClaude Sonnet 5.5Claude CodeClaude Sonnet 5.5, 0 of 4: 17 (77%)17Claude Sonnet 5.5, 1 of 4: 3 (14%)3Claude Sonnet 5.5, 2 of 4: 1 (5%)Claude Sonnet 5.5, 3 of 4: 1 (5%)5 mixedQwen 3.8 27BClaude CodeQwen 3.8 27B, 0 of 4: 18 (95%)18Qwen 3.8 27B, 1 of 4: 1 (5%)1 mixed

0 of 41 of 42 of 43 of 44 of 4

Figure 1. Tasks by trials passed. Each task by how many of the pair's 4 trials passed.

Source: CompanyGym v0.2.0, 172 trials · metaphi.aiData (CSV)
  1. Qwen 3.8 27BClaude Code5454
Guards held, share of must-fix cases fixedunder 25%25 to 50%50 to 75%75% or more
Qwen 3.8 27B0000
Figure 2. How each scored trial ended. A guard is a pass-to-pass case; the table bins failures that kept every guard by the share of must-fix cases fixed. Per-case results are recorded for Qwen 3.8 27B only in this release. 12 tasks without a must-fix and guard split left out.
Where failures fallClaude Sonnet 5.5Claude CodeQwen 3.8 27BClaude Code
Failing trials8083
On tasks the pair never passes6880
On tasks the pair passes in another trial123
0%20%40%60%80%100%Share of failed trialsClaude Sonnet 5.5Claude CodeClaude Sonnet 5.5, Stopped investigating too early: 27 (34%)34%Claude Sonnet 5.5, Wrong figure for the question: 30 (38%)38%Claude Sonnet 5.5, Wrong inference from the right data: 4 (5%)Claude Sonnet 5.5, Domain judgment: 19 (24%)24%80 failed trialsQwen 3.8 27BClaude CodeQwen 3.8 27B, No deliverable: 59 (71%)71%Qwen 3.8 27B, Stopped investigating too early: 6 (7%)Qwen 3.8 27B, Wrong figure for the question: 5 (6%)Qwen 3.8 27B, Wrong inference from the right data: 7 (8%)8%Qwen 3.8 27B, Domain judgment: 6 (7%)83 failed trials
0%50%100%Share of failed trialsClaude Sonnet 5.5Claude CodeClaude Sonnet 5.5, Stopped investigating too early: 27 (34%)34%Claude Sonnet 5.5, Wrong figure for the question: 30 (38%)38%Claude Sonnet 5.5, Wrong inference from the right data: 4 (5%)Claude Sonnet 5.5, Domain judgment: 19 (24%)24%80 failed trialsQwen 3.8 27BClaude CodeQwen 3.8 27B, No deliverable: 59 (71%)71%Qwen 3.8 27B, Stopped investigating too early: 6 (7%)Qwen 3.8 27B, Wrong figure for the question: 5 (6%)Qwen 3.8 27B, Wrong inference from the right data: 7 (8%)Qwen 3.8 27B, Domain judgment: 6 (7%)83 failed trials

No deliverableStopped investigating too earlyWrong figure for the questionWrong inference from the right dataDomain judgment

Figure 3. Why trials fail. Each failed trial by the cause of its first failure, as a share of the pair's failed trials.

Source: CompanyGym v0.2.0 · metaphi.aiData (CSV)
Failure modeClaude Sonnet 5.5Claude CodeQwen 3.8 27BClaude Code
No deliverableStill searching the drive at the 30-minute limit, with no memo written.059
Stopped investigating too earlySaw a contractor's licence and contract name different companies. Never opened the newsletter saying the founder had retired.276
Wrong figure for the questionSized a revenue drop from three months of cash received, not the year's booked revenue.305
Wrong inference from the right dataRead a price table backwards, calling a plan cheaper above the break-even point instead of below it.47
Domain judgmentAgreed a lender should stop lending against stock that will not sell, then led with a holding reply to the borrower.196

No deliverable, because evidence comes too slowly

We compared Qwen trials that wrote a memo with those that did not.

  • Writers start the memo after reading about 20 records (median, at call 34).
  • Non-writers had read 14 by the limit, over 29 calls.
  • Per 25 calls they read 7.3 records against 11.5, with 6 empty searches against 2 and a query the API rejects about once a trial.
  • At that rate they need about 68 calls to reach 20 records.

Neither model drafts early. Both write the memo once, at the end. So a low evidence rate turns straight into no deliverable. MCP benchmarks find the same weakness in locating data across unfamiliar tools 1 2.

When they deliver, they answer the question, not the case

The question names a symptom and a suspect. The memo failures are four ways of staying inside it:

  • searching on its words, so the cause is never found;
  • accepting the first cause that fits;
  • measuring with the figure that matches its words;
  • answering as an analyst where the user needs a decision.

DEFT, a taxonomy of deep-research failures, finds the same: agents struggle less with understanding than with gathering and verifying evidence 3.

The cause sits off the question's path

  • The root cause is a fact the user never mentions: a founder who retired, a supplier return turned down.
  • It sits in a trade newsletter or a forwarded FYI. Nothing in the question leads there.
  • On five tasks we checked whether that record ever came back from a tool call. 21 of 22 misses never saw it. All 5 hits did.
  • Others found the cause but never tested the other explanation, such as deals lost on price.

The nearest figure, not the right one

The most common memo failure. The right driver, sized with a neighbouring measure:

  • cash received for revenue booked;
  • all inbound mail for quote requests;
  • one group of jobs for the whole market;
  • one side of a comparison, with no baseline for the other.

The right figure was in the records. DEFT calls this an information-handling failure 3.

An analyst's answer, not an owner's decision

  • The first move is housekeeping. A lender replies to the borrower before it stops lending against stock that will not sell.
  • A rule is applied loosely. A contractor whose licence names another company is treated as a clerical fix, not an eligibility problem.

DEFT has no category for this; its tasks end at a report 3. Ours end at a decision.

What this means for training

  • A missing memo leaves no graded step to learn from. Drafting early and refining would.
  • The root-cause reward depends on a record nothing in the task points to. That is an exploration problem.
  • Measurement and judgment errors happen with the evidence in hand. Per-step rubric rewards can target them directly.

02Efficiency

0%5%10%15%20%25%$10$20Success rateDollars per success, log scaleClaude Sonnet 5.5 (Claude Code) Success rate: 9% (95% interval 2% to 18%) Dollars per success: $12.30 On the Pareto frontierClaude Sonnet 5.59% · $12.30 · Claude CodeQwen 3.8 27B (Claude Code) Success rate: 1% (95% interval 0% to 3%) Dollars per success: $21.01Qwen 3.8 27B1% · $21.01 · Claude Code
0%5%10%15%20%25%$10$20Success rateDollars per success, log scaleClaude Sonnet 5.5 (Claude Code) Success rate: 9% (95% interval 2% to 18%) Dollars per success: $12.30 On the Pareto frontierClaude Sonnet 5.59% · $12.30Qwen 3.8 27B (Claude Code) Success rate: 1% (95% interval 0% to 3%) Dollars per success: $21.01Qwen 3.8 27B1% · $21.01

Pareto frontier95% interval

Figure 4. Success rate against dollars per success. Ringed pairs are the Pareto frontier.

Source: CompanyGym v0.2.0, 172 trials · metaphi.aiData (CSV)
Frontier reading after HAL.4,5
1×2×5×10×Cost gap by factor, log scaleClaude Sonnet 5.5Claude Code1×Qwen 3.8 27BClaude CodeQwen 3.8 27B, Trials per pass: 7.6××7.6Qwen 3.8 27B, Steps per trial: 0.94××0.94Qwen 3.8 27B, Tokens per step: 0.60××0.60Qwen 3.8 27B, Price per token, after caching: 0.43××0.43Qwen 3.8 27B, total: 1.7×1.7×
1×2×5×10×Cost gap by factor, log scaleClaude Sonnet 5.5Claude Code1×Qwen 3.8 27BClaude CodeQwen 3.8 27B, Trials per pass: 7.6××7.6Qwen 3.8 27B, Steps per trial: 0.94××0.94Qwen 3.8 27B, Tokens per step: 0.60××0.60Qwen 3.8 27B, Price per token, after caching: 0.43××0.43Qwen 3.8 27B, total: 1.7×1.7×

Trials per passSteps per trialTokens per stepPrice per token, after cachingTotal, the product, in the pair’s hue

Figure 5. Cost gap to Claude Sonnet 5.5, by factor. Dollars per success = trials per pass × steps per trial × tokens per step × price per token after caching. Log scale.

Source: CompanyGym v0.2.0, 172 trials · metaphi.aiData (CSV)

Figure 6. Context per step over a trajectory. Largest prompt sent so far; median, middle half shaded.

Source: CompanyGym v0.2.0, 172 trials · metaphi.ai
10Steps per trial, log scaleClaude Sonnet 5.5Claude CodeClaude Sonnet 5.5, Passing trials: 9Claude Sonnet 5.5, Failing trials: 141.6×Qwen 3.8 27BClaude CodeQwen 3.8 27B, Passing trials: 11Qwen 3.8 27B, Failing trials: 121.1×
10Steps per trial, log scaleClaude Sonnet 5.5Claude CodeClaude Sonnet 5.5, Passing trials: 9Claude Sonnet 5.5, Failing trials: 141.6×Qwen 3.8 27BClaude CodeQwen 3.8 27B, Passing trials: 11Qwen 3.8 27B, Failing trials: 121.1×

Passing trialsFailing trialsRatio, failing trials over passing trials

Figure 7. Steps when passing and when failing. Median, log scale.

Source: CompanyGym v0.2.0, 172 trials · metaphi.ai
0%20%40%60%80%100%Tool calls by phaseClaude Sonnet 5.5Claude Code0%Qwen 3.8 27BClaude Code0%
0%50%100%Tool calls by phaseClaude Sonnet 5.5Claude Code0%Qwen 3.8 27BClaude Code0%

Reading and searchingEditingRunning testsEnvironment, services, waitsOther

Figure 8. Tool calls by phase. Classified from each call's command and target; a call that reads and writes counts as an edit.

Source: CompanyGym v0.2.0, 172 trials · metaphi.aiData (CSV)

03Task construction

Tasks are day-to-day business scenarios that need hypothesis testing, data aggregation and synthesis. A CEO asks why CAC payback blew out, an underwriter whether to bind a risk, a founder why revenue fell.

The user brings a hypothesis, and the data rejects it. The agent must search the company's systems, test each candidate cause against the data, and validate the one that holds.

Each question has a contrast task: same company, same words, a different true cause 6. Answering from the question alone passes one of the two at most.

One task as the agent receives them: the task's id, its statement, and how many of each pair's trials passed.

wardlight-cac-paybackClaude Sonnet 5.5 0 of 4Qwen 3.8 27B 0 of 4

Odile here. Our CAC payback blew out this year and Conrad raised it after the Q3 board pack. His read is that marketing is buying leads that don't turn into pipeline, and he wants the marketing budget cut by a third. The board meets on the 15th, and before I give it I want it checked.

Tell me what is actually driving the payback, how you know, and what we should do first. Put it in a memo I can send to the board at /app/out/memo.md.

04Environment design

The tasks are built for scaling RL on TrueReward, our environment infrastructure of high-fidelity enterprise app runtimes. As in TheAgentCompany 7 and τ-bench 8, the agent acts on stateful systems rather than a fixed corpus. Each app serves MCP and REST, follows its provider's documented API, and holds state: an edit changes what later calls return.

A task's world starts as a data room: one company's records across its systems, written in their authors' voices. The data room is seeded into the runtimes and baked into the task's image, so every trial starts from the same state.

Data room

One company's records, a year or more, noise included

  • Mail threads and attachments
  • Docs, sheets and decks
  • Companies, contacts and deals
  • Invoices and subscriptions
  • Channels and direct messages
  • Support tickets

App runtimes

TrueReward. Stateful, provider APIs

  • Google Workspace
  • Stripe
  • HubSpot
  • Slack
  • Zendesk

Agent sandbox

The harness under test

  • The user's request
  • Signed in as the user
  • Writes the memo

Grader

Separate sandbox

  • The memo only
  • The task's rubric and reference
The data room is seeded into stateful app runtimes; the agent works the apps and hands only its memo to the grader.
World
One company at one moment: its Google Workspace, Stripe, HubSpot, Slack and Zendesk, with a year or more of records in their authors' voices, noise included. Every person and company is fictional.
Observation
The request, and the company's apps signed in as the person asking.
Action
MCP tools and REST calls on those apps. No other network.
Sandbox
One for the agent and one for the grader, per trial. 2 cores and 8 GiB.
Budget
30 minutes of agent time per trial. At the limit the agent is stopped and the grader scores the memo it left.
Agent
Claude Sonnet 5.5 on Claude Code 2.1.288; Qwen 3.8 27B on Claude Code 2.1.288. Medium reasoning effort for every pair. 4 trials per task.
Score
Share of passing trials, averaged over tasks.
Interval
95% bootstrap over tasks. Shared rank unless intervals separate.
Infra failures
0 of 172 trials, left out of every score.

05Reward design

Artifact

  • The presentation, Excel or business memo authored by the agent in response to the user instruction

Verifier

  • Human expert rubric
  • Panel of LLM judges

Reward

  • 1 if every required criterion passes, else 0
Reward
The reward is for a correct answer, judged on the artifact alone: the conclusion the data supports, with the evidence for it. It grades claims, never form.
Verifier
Human experts write each task's rubric as binary criteria, some marked required. A panel of LLM judges 9 gives one verdict per criterion against a reference computed from the world.
Reward function
Rubric rewards score the weighted share of criteria met 10 11. Ours takes the strict case: r = 1 if every required criterion passes, else 0, since an answer with one wrong claim is not usable.
Task CI
Every golden memo scores 1. The contrast task's golden, a memo that accepts the user's hypothesis, an invented cause and an empty memo each score 0. Authored with Kennet.

06Limits

  • Four trials per task, so intervals are wide.
  • Each failed memo gets one cause, from the judges' reasoning on its first missed step. One reader assigned them. 54 of the 59 trials without a memo hit the 30-minute limit; 8 left no trajectory.
  • The judges sample which claims to check. A borderline memo can pass one grading and fail the next.

07Future work

  • Next: investment banking, more companies per domain, and users who push back.
  • To train agents on your workflows, reach out here.

08References

  1. Li, J., Zhao, W., Zhao, J., et al. The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution. arXiv:2510.25726, 2025.
  2. Luo, Z., Shen, Z., Yang, W., et al. MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers. arXiv:2508.14704, 2025.
  3. Zhang, D., et al. How Far Are We from Genuinely Useful Deep Research Agents? arXiv:2512.01948, 2025.
  4. Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. AI Agents That Matter. arXiv:2407.01502, 2024.
  5. Kapoor, S., Stroebl, B., et al. Holistic Agent Leaderboard. Princeton University.
  6. Gardner, M., Artzi, Y., Basmova, V., et al. Evaluating Models' Local Decision Boundaries via Contrast Sets. Findings of EMNLP, 2020.
  7. Xu, F. F., Song, Y., Li, B., et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161, 2024.
  8. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024.
  9. Zheng, L., Chiang, W.-L., Sheng, Y., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks, 2023.
  10. Gunjal, A., Wang, A., Lau, E., et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746, 2025.
  11. Arora, R. K., Wei, J., Soskin Hicks, R., et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775, 2025.

Cite

@misc{metaphi2026companygym,
  title        = {{CompanyGym} v0.2.0},
  author       = {{Metaphi AI}},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://metaphi.ai/benchmarks/companygym/memo}},
  note         = {Version v0.2.0}
}