MainframeGym

01Environment

Mainframes still carry the core ledgers of carriers, banks and insurers, in COBOL and JCL written over decades. A business rule there lives across many batch programs and the jobs that chain them, and a change is correct only when the business outcome of every job it touches is right. MainframeGym places an agent in such an estate with a request worded as the business makes it, and grades the outputs of the jobs it changed.

System

A monolithic billing estate for a telephone carrier: a median of 123 production COBOL programs (175k lines), some 200 JCL members, 53 copybooks and 177 Db2 tables. Each task starts from the estate at its own point in history, so the tree differs by task: 228k to 410k lines of COBOL, JCL, copybooks and DDL.

Observation

The full source tree, the task statement, and sample input data staged for the agent's own runs. The statement names the business outcome, never the program, the job or the field.

Action

A shell in the agent's container. harwell submit <deck> compiles what the deck needs, runs the batch job on Harwell and returns its report: every job and step, return code, printed output and output dataset.

Hidden

The cases: input datasets and Db2 rows, the job and step each case grades, and the expected outputs. The reference fix. The sample data is never built from the cases.

Budget

Two hours of wall clock per trial. No network. A trial that runs out of time scores 0.

Reward

Binary. r = 1 when every fail-to-pass case and every pass-to-pass case passes, else 0. A case passes when the output records, Db2 rows and report counters its graded step writes match the expected ones. Any implementation with that behaviour passes.

Grader

A separate container. It compiles the changed programs, runs each case's batch job on Harwell from that case's own input, and checks the business outcome, never the source text. Fail-to-pass cases test the requested change; pass-to-pass cases pin billing that must not move.

Task CI

Each defect lives in the estate's own programs. Task CI runs every known-wrong fix (must fail) and every known-right fix (must pass). Authored with Kennet.

Environment

A mainframe runtime for batch and online work, with its divergences from IBM published. No network.

Agent

Claude Opus 5.5 on Claude Code 2.1.288, medium effort; Gemini 4 Argon on Antigravity 1.3.5, high effort; Gemini 3.8 Flash on Terminus 2 2.0.0, medium effort. 4 trials per task.

Score

Share of passing trials, averaged over tasks.

Interval

95% bootstrap over tasks. Shared rank unless intervals separate.

02Runtime: Harwell

Harwell is our mainframe runtime. The agent and the grader run the same engine, so what the agent observes is what the reward reads.

A mainframe reward is only as sound as the runtime beneath it: COBOL packed and zoned decimal has truncation rules a stock compiler does not follow, and a runtime one cent off grades a correct fix as wrong. Harwell's patched GnuCOBOL 3.2 reproduces IBM numeric behaviour. Every run states its fidelity, clean or degraded with each divergence named against a published table; a case whose verdict would rest on one is refused at authoring.

Harwell runs code and reports. It grades nothing and knows no task; the grader reads its report.

Batch

JCL jobs and procedures, COBOL, DFSORT, IDCAMS, VSAM and generation data groups.

Data

Embedded SQL against the estate's Db2 schema, on SQLite through GixSQL.

Online

A CICS region with BMS maps, started transaction by transaction. Not exercised by this release.

Report

One JSON document per run. Each job and step, its return code, what it printed, and the fidelity of the run.

Evidence

Its readers are measured on every push against two public mainframe applications, AWS CardDemo and IBM's CICS GenApp, each pinned to a commit.

{
  "report": 1, "mode": "decks", "status": "ok",
  "jobs": [{
    "name": "HELLOJOB", "status": "ok", "worst_rc": 0,
    "steps": [{"name": "STEP01", "pgm": "HELLO", "kind": "cobol",
               "status": "ran", "rc": 0,
               "stdout": "HELLO FROM THE SANDBOX\n"}]
  }],
  "fidelity": {"status": "clean", "neutralized": [], "divergences": []},
  "metrics": {"cpu_s": 0.013, "wall_s": 0.981, "max_rss_kb": 2480}
}

03Task construction

Hidden cases

338 (183 fail-to-pass, 155 pass-to-pass). Median 18.5 per task.

Known-wrong

180 plausible wrong fixes, each a realistic mistake, that the verifier must reject. Median 10 per task. Every one is rejected.

Known-right

43 correct fixes besides the reference, written differently, that the verifier must accept; 5 written blind from the statement alone. Every one is accepted.

The three comparison columns are as DeepSWE reports them.³ Our reference is measured against the tree the agent starts from: the median task has a 1,016-character statement and a fix that adds 79 lines across 3 files.

Two of the tasks as the agent receives them: the task's id, a passage of its statement, and how many of each pair's trials passed.

04Results by axis

Success rate averaged over each row's tasks; best pair in bold. Average rank ranks the pairs on each task by pass rate, ties shared.

05Failure analysis

Claude Opus 5.5 on Claude Code passes 39 of 64 trials, Gemini 4 Argon on Antigravity 19 and Gemini 3.8 Flash on Terminus 2 8. Each pair gets two hours per trial.

I will run harwell submit TST03600 --from ../opt/start using a relative path to /opt/start.

Every account in this sample was already correct, so the defect doesn't show in it.

Running the scaled test job (SIU04425) to verify execution with 400 0800 numbers, 400 discount tiers, and 20,000 localities.

In SIU05420.cbl and SIU05110.cbl, 'OR '8'' was already added to TERORIG and TERDEST checks.

So Call 6 MATCHES promotion 201!

I didn't add an automatic purge because I couldn't scope it safely.

Therefore, I must not modify this critical section.

06Pareto frontier

2 tasks are solved by every pair, 4 by none.

07Training signal

08Future work

• Online work: CICS transactions and BMS screens, graded screen by screen.
• Performance: CPU and elapsed time per job as a continuous reward.
• Modernization: COBOL rewritten as Java, graded against the original estate.

09References

1. Chowdhury, N., Aung, J., Shern, C. J., et al. Introducing SWE-bench Verified. OpenAI, 2024.
2. Deng, X., Da, J., Pan, E., et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941, 2025.
3. Datacurve. DeepSWE. Datacurve blog.
4. Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. AI Agents That Matter. arXiv:2407.01502, 2024.
5. Kapoor, S., Stroebl, B., et al. Holistic Agent Leaderboard. Princeton University.
6. Erol, M. H., El, B., Suzgun, M., Yuksekgonul, M., and Zou, J. Cost-of-Pass: An Economic Framework for Evaluating Language Models. arXiv:2504.13359, 2025.
7. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024.