Takeaways
- The first failure is no deliverable. Qwen 3.8 27B wrote no memo in 70% of its trials: it gathers evidence too slowly to reach the point where it writes.
- Given the evidence, agents reason well. Wrong inference causes 4 of the best configuration's 80 failures.
- They fail on what they never look for. In 21 of 22 root-cause misses we traced, the deciding record was never opened.
- They size the right driver with the nearest figure, and stop at a diagnosis where the user needs a decision.
01Effectiveness
| Tasks | Claude Sonnet 5.5Claude Code | Qwen 3.8 27BClaude Code |
|---|---|---|
| All tasks | ||
| Success rate, pass@1 | 9% | 1% |
| Ceiling, pass@2 | 15% | 2% |
| Reliability, pass^2 | 3% | 0% |
| Avg rank over tasks1 is best | 1.39 | 1.61 |
0 of 41 of 42 of 43 of 44 of 4
Figure 1. Tasks by trials passed. Each task by how many of the pair's 4 trials passed.
- Qwen 3.8 27BClaude Code54
| Guards held, share of must-fix cases fixed | under 25% | 25 to 50% | 50 to 75% | 75% or more |
|---|---|---|---|---|
| Qwen 3.8 27B | 0 | 0 | 0 | 0 |
| Where failures fall | Claude Sonnet 5.5Claude Code | Qwen 3.8 27BClaude Code |
|---|---|---|
| Failing trials | 80 | 83 |
| On tasks the pair never passes | 68 | 80 |
| On tasks the pair passes in another trial | 12 | 3 |
No deliverableStopped investigating too earlyWrong figure for the questionWrong inference from the right dataDomain judgment
Figure 3. Why trials fail. Each failed trial by the cause of its first failure, as a share of the pair's failed trials.
| Failure mode | Claude Sonnet 5.5Claude Code | Qwen 3.8 27BClaude Code |
|---|---|---|
| No deliverableStill searching the drive at the 30-minute limit, with no memo written. | 0 | 59 |
| Stopped investigating too earlySaw a contractor's licence and contract name different companies. Never opened the newsletter saying the founder had retired. | 27 | 6 |
| Wrong figure for the questionSized a revenue drop from three months of cash received, not the year's booked revenue. | 30 | 5 |
| Wrong inference from the right dataRead a price table backwards, calling a plan cheaper above the break-even point instead of below it. | 4 | 7 |
| Domain judgmentAgreed a lender should stop lending against stock that will not sell, then led with a holding reply to the borrower. | 19 | 6 |
No deliverable, because evidence comes too slowly
We compared Qwen trials that wrote a memo with those that did not.
- Writers start the memo after reading about 20 records (median, at call 34).
- Non-writers had read 14 by the limit, over 29 calls.
- Per 25 calls they read 7.3 records against 11.5, with 6 empty searches against 2 and a query the API rejects about once a trial.
- At that rate they need about 68 calls to reach 20 records.
Neither model drafts early. Both write the memo once, at the end. So a low evidence rate turns straight into no deliverable. MCP benchmarks find the same weakness in locating data across unfamiliar tools 1 2.
When they deliver, they answer the question, not the case
The question names a symptom and a suspect. The memo failures are four ways of staying inside it:
- searching on its words, so the cause is never found;
- accepting the first cause that fits;
- measuring with the figure that matches its words;
- answering as an analyst where the user needs a decision.
DEFT, a taxonomy of deep-research failures, finds the same: agents struggle less with understanding than with gathering and verifying evidence 3.
The cause sits off the question's path
- The root cause is a fact the user never mentions: a founder who retired, a supplier return turned down.
- It sits in a trade newsletter or a forwarded FYI. Nothing in the question leads there.
- On five tasks we checked whether that record ever came back from a tool call. 21 of 22 misses never saw it. All 5 hits did.
- Others found the cause but never tested the other explanation, such as deals lost on price.
The nearest figure, not the right one
The most common memo failure. The right driver, sized with a neighbouring measure:
- cash received for revenue booked;
- all inbound mail for quote requests;
- one group of jobs for the whole market;
- one side of a comparison, with no baseline for the other.
The right figure was in the records. DEFT calls this an information-handling failure 3.
An analyst's answer, not an owner's decision
- The first move is housekeeping. A lender replies to the borrower before it stops lending against stock that will not sell.
- A rule is applied loosely. A contractor whose licence names another company is treated as a clerical fix, not an eligibility problem.
DEFT has no category for this; its tasks end at a report 3. Ours end at a decision.
What this means for training
- A missing memo leaves no graded step to learn from. Drafting early and refining would.
- The root-cause reward depends on a record nothing in the task points to. That is an exploration problem.
- Measurement and judgment errors happen with the evidence in hand. Per-step rubric rewards can target them directly.
02Efficiency
Pareto frontier95% interval
Figure 4. Success rate against dollars per success. Ringed pairs are the Pareto frontier.
Trials per passSteps per trialTokens per stepPrice per token, after cachingTotal, the product, in the pair’s hue
Figure 5. Cost gap to Claude Sonnet 5.5, by factor. Dollars per success = trials per pass × steps per trial × tokens per step × price per token after caching. Log scale.
Figure 6. Context per step over a trajectory. Largest prompt sent so far; median, middle half shaded.
Passing trialsFailing trialsRatio, failing trials over passing trials
Figure 7. Steps when passing and when failing. Median, log scale.
Reading and searchingEditingRunning testsEnvironment, services, waitsOther
Figure 8. Tool calls by phase. Classified from each call's command and target; a call that reads and writes counts as an edit.
03Task construction
Tasks are day-to-day business scenarios that need hypothesis testing, data aggregation and synthesis. A CEO asks why CAC payback blew out, an underwriter whether to bind a risk, a founder why revenue fell.
The user brings a hypothesis, and the data rejects it. The agent must search the company's systems, test each candidate cause against the data, and validate the one that holds.
Each question has a contrast task: same company, same words, a different true cause 6. Answering from the question alone passes one of the two at most.
One task as the agent receives them: the task's id, its statement, and how many of each pair's trials passed.
Odile here. Our CAC payback blew out this year and Conrad raised it after the Q3 board pack. His read is that marketing is buying leads that don't turn into pipeline, and he wants the marketing budget cut by a third. The board meets on the 15th, and before I give it I want it checked.
Tell me what is actually driving the payback, how you know, and what we should do first. Put it in a memo I can send to the board at /app/out/memo.md.
04Environment design
The tasks are built for scaling RL on TrueReward, our environment infrastructure of high-fidelity enterprise app runtimes. As in TheAgentCompany 7 and τ-bench 8, the agent acts on stateful systems rather than a fixed corpus. Each app serves MCP and REST, follows its provider's documented API, and holds state: an edit changes what later calls return.
A task's world starts as a data room: one company's records across its systems, written in their authors' voices. The data room is seeded into the runtimes and baked into the task's image, so every trial starts from the same state.
Data room
One company's records, a year or more, noise included
- Mail threads and attachments
- Docs, sheets and decks
- Companies, contacts and deals
- Invoices and subscriptions
- Channels and direct messages
- Support tickets
App runtimes
TrueReward. Stateful, provider APIs
- Google Workspace
- Stripe
- HubSpot
- Slack
- Zendesk
Agent sandbox
The harness under test
- The user's request
- Signed in as the user
- Writes the memo
Grader
Separate sandbox
- The memo only
- The task's rubric and reference
- World
- One company at one moment: its Google Workspace, Stripe, HubSpot, Slack and Zendesk, with a year or more of records in their authors' voices, noise included. Every person and company is fictional.
- Observation
- The request, and the company's apps signed in as the person asking.
- Action
- MCP tools and REST calls on those apps. No other network.
- Sandbox
- One for the agent and one for the grader, per trial. 2 cores and 8 GiB.
- Budget
- 30 minutes of agent time per trial. At the limit the agent is stopped and the grader scores the memo it left.
- Agent
- Claude Sonnet 5.5 on Claude Code 2.1.288; Qwen 3.8 27B on Claude Code 2.1.288. Medium reasoning effort for every pair. 4 trials per task.
- Score
- Share of passing trials, averaged over tasks.
- Interval
- 95% bootstrap over tasks. Shared rank unless intervals separate.
- Infra failures
- 0 of 172 trials, left out of every score.
05Reward design
Artifact
- The presentation, Excel or business memo authored by the agent in response to the user instruction
Verifier
- Human expert rubric
- Panel of LLM judges
Reward
- 1 if every required criterion passes, else 0
- Reward
- The reward is for a correct answer, judged on the artifact alone: the conclusion the data supports, with the evidence for it. It grades claims, never form.
- Verifier
- Human experts write each task's rubric as binary criteria, some marked required. A panel of LLM judges 9 gives one verdict per criterion against a reference computed from the world.
- Reward function
- Rubric rewards score the weighted share of criteria met 10 11. Ours takes the strict case: r = 1 if every required criterion passes, else 0, since an answer with one wrong claim is not usable.
- Task CI
- Every golden memo scores 1. The contrast task's golden, a memo that accepts the user's hypothesis, an invented cause and an empty memo each score 0. Authored with Kennet.
06Limits
- Four trials per task, so intervals are wide.
- Each failed memo gets one cause, from the judges' reasoning on its first missed step. One reader assigned them. 54 of the 59 trials without a memo hit the 30-minute limit; 8 left no trajectory.
- The judges sample which claims to check. A borderline memo can pass one grading and fail the next.
07Future work
- Next: investment banking, more companies per domain, and users who push back.
- To train agents on your workflows, reach out here.
08References
- Li, J., Zhao, W., Zhao, J., et al. The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution. arXiv:2510.25726, 2025.
- Luo, Z., Shen, Z., Yang, W., et al. MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers. arXiv:2508.14704, 2025.
- Zhang, D., et al. How Far Are We from Genuinely Useful Deep Research Agents? arXiv:2512.01948, 2025.
- Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. AI Agents That Matter. arXiv:2407.01502, 2024.
- Kapoor, S., Stroebl, B., et al. Holistic Agent Leaderboard. Princeton University.
- Gardner, M., Artzi, Y., Basmova, V., et al. Evaluating Models' Local Decision Boundaries via Contrast Sets. Findings of EMNLP, 2020.
- Xu, F. F., Song, Y., Li, B., et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161, 2024.
- Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024.
- Zheng, L., Chiang, W.-L., Sheng, Y., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks, 2023.
- Gunjal, A., Wang, A., Lau, E., et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746, 2025.
- Arora, R. K., Wei, J., Soskin Hicks, R., et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775, 2025.
Cite
@misc{metaphi2026companygym,
title = {{CompanyGym} v0.2.0},
author = {{Metaphi AI}},
year = {2026},
month = oct,
howpublished = {\url{https://metaphi.ai/benchmarks/companygym/memo}},
note = {Version v0.2.0}
}