Metaphi AI

Off the shelf

Coding

179tasksCoding taskstasksRepository before the fix, the ticket in the expert's words, a verifier that executes.executed verifier · binarypass@1 23%
Domains
enterprise software engineering, performance engineering, mainframe engineering
Signal
executed verifier · binary reward · task CI
Format
Harbor task bundle
Seed
history and expert scenarios
Expert
mainframe and Go engineers
Runtime
mainframe emulator and containers
Access
Available · sample · commercial
Frontier
pass@1 23% · Aug 2026
Counted
10 Sep 2026
3,600trajectories100 tasks coveredCoding agent trajectoriestrajectoriesEvery action, the reward and the verifier output, from six models under four harnesses.executed verifier · binarypass@1 14%
Domains
mainframe engineering
Signal
executed verifier · binary reward
Format
JSONL per trajectory with the verifier report
Seed
benchmark rollouts
Models
6
Harnesses
4
Access
Available · sample · commercial
Frontier
pass@1 14% · May 2026 · best model; 0.04 worst
Counted
10 Sep 2026
1.6Mlines1,114 programs · 5,257 copybooksMainframe estatescode repositoriesProduction COBOL, JCL and SQL on git, from telecom billing to core banking, every program runnable.executed verifier · binary
Domains
mainframe engineering
Signal
executed verifier · binary reward · a change is checked by execution
Format
git repository with history and enterprise context
Seed
production
Runtime
mainframe emulator
Access
On request · commercial
Counted
10 Sep 2026

Design

10tasks37 design filesFigma environmenttasksraw dataA headless Figma runtime with the Plugin API and MCP tools, real designs as deterministic documents.hierarchical verifier · scalar
Domains
design agents
Signal
hierarchical verifier · scalar reward · hard gates
Format
container per trial, Plugin API and MCP
Seed
expert authored
Expert
product designers
Runtime
Figma clone
Access
Available · sample · commercial
Counted
10 Sep 2026
96trajectories147 expert pairsFigma to codetrajectoriesreasoningAgents turning real designs into React and Tailwind, with the deployed output and expert pairwise judgments.expert preference · rankingElo 1,054
Domains
design agents
Signal
expert preference · ranking reward · reason and quality rating per vote
Format
JSONL trajectories, deployed previews, preference pairs
Seed
community designs
Expert
product designers
Harnesses
3
Access
Available · sample · commercial
Frontier
Elo 1,054 · Mar 2026 · top two agents tied
Counted
10 Sep 2026

Financial services

8,900PDFs5.6M labelled transactions · 1,387 casesBank statements to transactionsraw datatasksBusiness bank statement PDFs, every transaction labelled by type and counterparty by risk analysts.programmatic · scalarpass@1 33%
Domains
financial documents
Signal
programmatic · scalar reward · extraction F1 and categorization accuracy
Format
PDF with JSON ground truth, one container per case
Seed
production
Expert
risk analysts with ten or more years
PII
removed
Access
Available · sample · commercial
Frontier
pass@1 33% · Mar 2026
Counted
10 Sep 2026

Content

45courses183 chapters · 200 source documents · 279 expert pairsLongGroundtasksraw datareasoningCourses from source documents with line-level citations, rendered as programmatic video and judged by experts.rubric · scalar
Domains
content generation
Signal
rubric · scalar reward · citation grounding and rubrics per chapter
Format
JSON per course, React video per chapter, preference pairs
Seed
expert authored
Expert
professional educators
Access
Available · sample · commercial
Counted
10 Sep 2026

Task format

instruction.md    the task in the expert's words
task.toml         metadata, timeouts, resources
environment/      Dockerfile, the repository at its starting commit, tools
solution/         the reference solution
tests/            the verifier and its cases

Post training