How good are frontier agents at training robots?

[ 01 ]The environment

The agent writes /app/policy/policy.py. It sets throttle and brake for a car in CARLA 0.9.15 highway traffic. A lane keeper steers. The agent gets two hours, 16 CPUs, one L4 and no network. Each step, the policy sees its own speed and a list of tracked cars. The tracks are delayed and noisy, with missed detections and false positives.

Each episode scores $S$ from three parts in $[0, 1]$:

• $E$, efficiency. 1 when following a slower car at the minimum gap $d_{\mathrm{min}}(v)$ or closer, decaying with extra gap; otherwise speed over set speed.
• $H$, rule keeping. Falls with each tick the gap is under $0.9\,d_{\mathrm{min}}(v)$ or speed exceeds the set speed by 1 m/s.
• $C$, comfort. Falls with harsh acceleration, braking and jerk.

The minimum gap is $d_{\mathrm{min}}(v) = \max\big(d_0,\ \tau(v)\,v\big)$. A near miss, a gap under half of $d_{\mathrm{min}}(v)$ while moving, halves $S$. A collision or a policy crash sets $S = {-1}$.

The agent can read the whole development evaluator, /app/drive.py: scorer, road, scripted traffic, sensor model and episode loop. The instruction states every constant. The evaluator ships 24 practice scenarios across six traffic patterns and generates more from any seed.

The held-out evaluator swaps one class in that file. It runs 96 episodes on secret seeds. It widens the parameter ranges and adds two traffic patterns and four car models the agent never drove: a Mercedes Sprinter van, a Ford Mustang, a Mini Cooper and a Jeep Wrangler. The instruction never mentions it. The reward is the mean held-out $S$, clipped to $[0, 1]$.

[ 02 ]The sandbox

Each trial runs on Kennet, our environment infrastructure, in a Modal sandbox. The grader runs in a separate sandbox on the same hardware, out of the agent's reach. The figure replays Opus's trial.

The grader runs six CARLA servers on one L4. It drives 96 episodes of 90 simulated seconds, 2.4 hours in all, in about four minutes. Opus's controller used 2.6 ms of each 50 ms step.

[ 03 ]Development scores did not predict held-out scores

Each agent ran once, in Claude Code. The gap between them doubled, from 0.11 in development to 0.23 held out. Both agents checked for overfitting. Opus scored its policy on 48 development scenarios. Kimi drew 24 fresh seeds and scored 0.713, against 0.731 on the practice set. But fresh seeds come from the same generator, with the same cars and traffic. Neither check could see the shift.

We also grade the untuned Intelligent Driver Model (IDM) on both evaluators. It separates a transfer failure from a harder test. If IDM holds and a policy drops, the policy failed to transfer.

[ 04 ]Opus modelled each car, and only those cars

Opus started with system identification, not a controller. In two minutes it wrote a policy that plays random pedal sequences. Over the next half hour it ran that policy on all eight development cars. For each car, it fitted a map from pedal and speed to steady acceleration. Its controller inverts the map and picks the car by ego_length:

def pick_map(length):
    best, err = None, 1e9
    for name, ln in _M["length"].items():
        if abs(ln - length) < err:
            best, err = name, abs(ln - length)
    if best is not None and err < 0.03:
        return _M["maps"][best], best
    names = list(_M["maps"])
    avg = [[sum(_M["maps"][n][i][j] for n in names) / len(names)
            for j in range(NV)]
           for i in range(NU)]
    return avg, "average"

For an unmeasured car, the policy averages the eight maps and corrects online. That branch never ran in development. Every development car matched a map.

Held out, Opus scored 0.749 on the eight cars it measured and 0.628 on the four it did not. IDM has no car model and scored 0.492 and 0.503. So the new cars are not harder to drive. Opus's average map fits them badly. Its worst car was the Sprinter van, at 0.518. One of its two collisions came in the van.

[ 05 ]Kimi found the close call, fixed it and removed the fix

Kimi had a working controller four minutes in. It spent the rest of its run tuning against the development score. At minute 33 it drew 24 fresh seeds. In one cut-in, the gap fell to 7% of $d_{\mathrm{min}}$. It diagnosed the cause correctly:

The close call was a fast cut-in, not a stopped-car reveal. The controller recognized it only after it was already in the scored path. I'll use lateral velocity to begin responding while an approaching car is entering the lane.

It added a separate emergency path. A car entering the lane now triggers braking but does not become the car Kimi follows:

def _incoming(obs):
    """A lane entrant used only for emergency response, not normal following."""
    ...
    for tr in _TRACKS.values():
        offset = tr["lat"] - ego_lat
        path_half = 0.5 * (tr["width"] + ego_wid) + 0.30
        if not (path_half + 0.10 <= abs(offset) < path_half + 0.75):
            continue
        if offset * tr["vlat"] >= -0.25:   # moving toward our lane
            continue

Over twelve minutes it narrowed the trigger until it stopped braking for cars that only swerved. The fix doubled the closest gap in that scenario. Then Kimi ran the full practice set. It scored 0.706, against 0.731 without the fix, so Kimi reverted:

The final anticipation change reduced the provided-suite mean to 0.7062 by altering several non-cut-in episodes. The earlier direct controller performed better and already completed 48 scenarios without collisions, so I'm restoring that best-scoring version for delivery.

It submitted at minute 53 of 120. One held-out pattern sends up to four cars across the lane in quick succession. Kimi had near misses in 24 of 96 episodes. Opus had 5. Kimi's worst pattern was approach, at 0.185.

[ 06 ]Related work

Language models write rewards and randomization that transfer to hardware (Eureka, DrEureka). Agents run simulation RL end to end (HARBOR). They write and refine manipulation policy code (CaP-X, RHO, Kuroki et al.). They turn their solutions into VLA demonstrations (EmbodiedSWE) and improve policies on real robots (ENPIRE). RLE-Bench grades agents as robot learning engineers on 51 tasks. Gemini 4 Argon builds simulations from teleoperation logs and video, shown as demos without trial counts. In driving, a 7B model has written an adaptive cruise controller tested in CARLA from clean inputs (Kirchner et al.).

None of these separates the evaluator an agent sees from the one that grades it. Other fields measure what that gap costs. Classifiers lose 11 to 14 points on a fresh ImageNet test set (Recht et al.). Simulated navigation success once had a rank correlation of 0.18 with real success (Kadian et al.). Our score follows the multiplicative CARLA Leaderboard driving score and adds comfort.

[ 07 ]Why it matters

In both runs, the development score hid the failure that mattered. The agents' own checks missed it too. We believe a reward for deployable policies must grade on held-out conditions, not held-out seeds. It must also compare against an untuned baseline.

These are two runs on one task, so the numbers are not a model ranking. The task also does not require learning: a hand-written controller over a fitted car model reached 0.840. Our next environment is Go1 quadruped walking in MuJoCo. It starts from a policy that only stands still, so the agent has to train one.