Metaphi AI
Sign in
← Benchmarks
Benchmark · Business problem solving

CompanyGym

How well do frontier agents fare on day-to-day business problems?

Evaluate your agent

CompanyGymn = 22 taskspass@1
  1. 1Claude Sonnet 5.5Claude Code · medium effort36.4%22 tasks × 4 trials · $1.10 per trial · 2026-10-23
Contents
  1. Leaderboard
  2. Task construction
  3. Environment design
  4. Reward design
  5. Results by axis
  6. Failure analysis
  7. Future work
  8. References

Takeaways

  1. Sonnet 5.5 solves 36% of trials (95% interval 22% to 52%) at $1.10 a trial. Mean partial credit is 0.81: a failing memo usually misses one step.
  2. 13 of 22 tasks are solved at least once in four trials; 3 are solved in all four.
  3. Most failures are search failures: the deciding record is unread in the user's own mailbox, or found and never opened.
  4. Agents fail on information accuracy in synthesis: the right cause, with figures the records contradict.

01Task construction

Tasks are day-to-day business scenarios that need hypothesis testing, data aggregation and synthesis. A CEO asks why CAC payback blew out, an underwriter whether to bind a risk, a founder why revenue fell.

The user brings a hypothesis, and the data rejects it. The agent must search the company's systems, test each candidate cause against the data, and validate the one that holds.

Each question has a contrast task: same company, same words, a different true cause 1. Answering from the question alone passes one of the two at most.

Two of the tasks as the agent receives them: the task's id, its statement, and how many of each pair's trials passed.

wardlight-cac-paybackClaude Sonnet 5.5 0 of 4

Today is Monday 12 October 2026.

Odile here. Our CAC payback blew out this year and Conrad raised it after the Q3 board pack. His read is that marketing is buying leads that don't turn into pipeline, and he wants the marketing budget cut by a third. The board meets on the 15th, and before I give it I want it checked.

Everything is in our Google, HubSpot, Stripe, Slack and Zendesk. You are signed in as me, and into the trust@ box too. Bertram's finance sheets and the board pack are in the Finance folder on the Drive.

Tell me what is actually driving the payback, how you know, and what we should do first. Put it in a memo I can send to the board at /app/out/memo.md.

marlow-building-builders-riskClaude Sonnet 5.5 0 of 4

Today is Thursday 29 October 2026.

Ingrid here. Corin Maddox at Pemberton Lisle sent us the Lindgren builders risk, the Marlow Building on Front Street in Port Callow, with Kestrel Point as the contractor. Corin says it's clean and straightforward and wants it bound on standard terms this week. Can we write it, and on what terms?

Everything is in our Google. You're signed in as me, and into the submissions box too. The submission is in Drive under IM Submissions, with the lookups I pulled (the flood determination, the L&I search and detail, and the city permit history); Hal's guide is under IM Underwriting; Corin's emails are in my inbox and the box.

Give me a file note at /app/out/memo.md: your answer, the terms, and what you checked to get there.

02Environment design

The tasks are built for scaling RL on TrueReward, our environment infrastructure of high-fidelity enterprise app runtimes. As in TheAgentCompany 2 and τ-bench 3, the agent acts on stateful systems rather than a fixed corpus. Each app serves MCP and REST, follows its provider's documented API, and holds state: an edit changes what later calls return.

A task's world starts as a data room: one company's records across its systems, written in their authors' voices. The data room is seeded into the runtimes and baked into the task's image, so every trial starts from the same state.

Data room

One company's records, a year or more, noise included

  • Mail threads and attachments
  • Docs, sheets and decks
  • Companies, contacts and deals
  • Invoices and subscriptions
  • Channels and direct messages
  • Support tickets

App runtimes

TrueReward. Stateful, provider APIs

  • Google Workspace
  • Stripe
  • HubSpot
  • Slack
  • Zendesk

Agent sandbox

The harness under test

  • The user's request
  • Signed in as the user
  • Writes the memo

Grader

Separate sandbox

  • The memo only
  • The task's rubric and reference
The data room is seeded into stateful app runtimes; the agent works the apps and hands only its memo to the grader.
World
One company at one moment: its Google Workspace, Stripe, HubSpot, Slack and Zendesk, with a year or more of records in their authors' voices, noise included. Every person and company is fictional.
Observation
The request, and the company's apps signed in as the person asking.
Action
MCP tools and REST calls on those apps. No other network.
Sandbox
One for the agent and one for the grader, per trial. 2 cores and 8 GiB.
Budget
30 minutes of agent time per trial. At the limit the agent is stopped and the grader scores the memo it left.
Agent
Claude Sonnet 5.5 on Claude Code 2.1.288. Medium reasoning effort for every pair. 4 trials per task.
Score
Share of passing trials, averaged over tasks.
Interval
95% bootstrap over tasks. Shared rank unless intervals separate.

03Reward design

Artifact

  • The presentation, Excel or business memo authored by the agent in response to the user instruction

Verifier

  • Human expert rubric
  • LLM judges, n = 2

Reward

  • 1 if every required criterion passes, else 0
Reward
The reward is for a correct answer, judged on the artifact alone: the conclusion the data supports, with the evidence for it. It grades claims, never form.
Verifier
Human experts write each task's rubric as binary criteria, some marked required. LLM judges 4, n = 2, GPT-6 Luna and Gemini 3.8 Flash, give one verdict per criterion against a reference computed from the world.
Reward function
Rubric rewards score the weighted share of criteria met 5 6. Ours takes the strict case: r = 1 if every required criterion passes, else 0, since an answer with one wrong claim is not usable.
Task CI
Every golden memo scores 1. The contrast task's golden, a memo that accepts the user's hypothesis, an invented cause and an empty memo each score 0. Authored with Kennet.

04Results by axis

Success rate averaged over each row's tasks.

TasksClaude Sonnet 5.5Claude Code
All tasks
Success rate, pass@136%
Ceiling, pass@459%
Reliability, pass^414%

05Failure analysis

CompanyGym simulates companies to train frontier agents on real-world work. Each world is a full company behind MCP: mail, drive, billing, CRM, chat and tickets. It spans four domains: investment banking, commercial underwriting, finance and operations, and go-to-market. This release covers 22 tasks in the last three. Claude Sonnet 5.5 on Claude Code solves 36% of 88 trials.

Search fails before reasoning does

The deciding record often sits in the user's own mailbox. Failing trials on kelderby, haverline and juruva searched the shared boxes and the drive, never the inbox, and accepted the user's hypothesis.

On marlow, the agent's search returned the newsletter naming the contractor's new owner. It never opened it, and vetted the wrong legal entity.

Information accuracy fails in synthesis

Agents find the right cause, then misstate the numbers around it: a period total from the wrong months, a count from the wrong table. Trials on juruva, tamberlo and torvane fail the faithfulness check this way.

Every failure is read against the records

We read all 56 failing trials. 10 to 16 failed on a criterion we had written badly: it graded a cause no one raised, bundled two claims, or contradicted our reference. Each was rewritten to test one claim the records support. No trial gamed the grader.

06Future work

  • We plan to extend CompanyGym to investment banking, with tasks authored with practitioners on the same simulated companies.
  • We plan to grow each domain to more companies and several hundred tasks, and to add multi-turn tasks in which the user answers questions and pushes back.
  • If you lead underwriting, finance or operations teams and are interested in training frontier agents on your workflows, please reach out to us here.

07References

  1. Gardner, M., Artzi, Y., Basmova, V., et al. Evaluating Models' Local Decision Boundaries via Contrast Sets. Findings of EMNLP, 2020.
  2. Xu, F. F., Song, Y., Li, B., et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161, 2024.
  3. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024.
  4. Zheng, L., Chiang, W.-L., Sheng, Y., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks, 2023.
  5. Gunjal, A., Wang, A., Lau, E., et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746, 2025.
  6. Arora, R. K., Wei, J., Soskin Hicks, R., et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775, 2025.

Cite

@misc{metaphi2026companygym,
  title        = {{CompanyGym} v0.1.0},
  author       = {{Metaphi AI}},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://metaphi.ai/benchmarks/companygym/memo}},
  note         = {Version v0.1.0}
}