Scaling compute on tacit knowledge
We propose a multi-loop harness that consults human experts as it designs environments and builds tasks, allowing us to scale task construction beyond the hours experts can spend hand authoring.
The scaling question is how much useful environment construction each hour of expert attention can support.
Access to human expertise limits progress at a growing frontier for models. Practitioners know which exceptions matter, which dependencies a change will touch, and when a correct-looking result would fail in practice. They have documented only part of that knowledge. Asking them to author every task, reconstruct its setup, and write its checks limits environment construction to their available hours.
We propose a multi-loop harness that consults human experts as it designs environments and builds tasks. We refer to it internally as Kennet. The harness explores the system, constructs candidate tasks, verifies them, and runs models against them. It consults experts on domain rules and task validity, then uses their answers to revise tasks and guide subsequent construction.
Why consultation matters
Experts explain why a scenario is unrealistic or a fix is unacceptable more readily than they enumerate every rule they use. The harness examines the system, traces its history, constructs a candidate, and presents the unresolved decision with evidence.
Consider a hypothetical billing adjustment after an invoice has closed. An expert might explain that the original invoice must remain unchanged and the adjustment must appear in a later cycle. The harness can use that rule to construct the setup and checks, then explore related cases involving timing, reversals, or downstream reconciliation.
The harness must validate each variation and consult the expert again when a new case raises a question the earlier answer does not resolve.
Three loops around task construction
The harness organizes the process around three loops: Check, Conduct, and Consult. They operate at different cadences.
The inner loop checks construction. The harness prepares the environment and tests whether the task and verifier behave as intended: a reference solution succeeds, doing nothing fails where a change is required, and the setup does not expose the answer. The harness revises candidates that fail these checks.
The middle loop runs models against the candidate to measure difficulty. The harness examines their attempts to distinguish demanding dependencies from ambiguous instructions, weak verifiers, or broken environments. It uses those findings to revise tasks and select further candidates.
The outer loop consults experts about domain questions, candidate realism, and the direction of exploration. The harness brings a concrete question with its context and continues independent construction while the expert considers it. Experts advise the environment design process separately from the agent attempt under evaluation.
The harness records each question, its context, the expert's answer, and unresolved issues in memory for subsequent sessions. It also proposes changes to task-authoring instructions and verifier rubrics for expert review. We then apply a correction to both the current task and the guidance for constructing future tasks.
Building the flywheel
We aim to build a flywheel from real work: a repository and its history, a ticket, or an expert session. Construction and model attempts reveal gaps; expert answers improve the tasks, checks, and guidance for the next round.
As models improve, their attempts provide new evidence about where the environment should expand. The harness can explore longer dependencies, additional exceptions, and combinations of constraints that arise in the underlying system. Experts help judge whether those extensions preserve the character of the work. This keeps task difficulty connected to real complexity.
Our mainframe work provides a concrete setting for this approach. Maintenance spans programs, data layouts, batch jobs, and business conventions. Reconstructing an executable task takes substantial mechanical work, while deciding whether it represents a meaningful problem requires someone who knows the system. The harness lets us automate much of the construction around that judgment.
We have implemented consultation, memory, and task calibration in the harness. We still need to measure whether reusing expert knowledge increases the yield of valid, diverse tasks per expert hour, and whether training on those tasks improves models on held-out work. Task count and model failure rates alone do not establish that result.
We want to scale the exploration and construction each expert contribution supports. By reusing their judgments across tasks and consulting them as new questions arise, we aim to build an expanding set of difficult environments that reflect the complexity of real work.