MA-Bench · internal preview · 2026-05-11

MA-Bench. A benchmark and a scaffold for meta-agents.

Eight capability directions. Three state primitives. One offline-trajectory-as-fixture mechanism that cancels worker variance. Same-system counterfactual contract throughout.

Press → / space to advance · ← to go back · or scroll

00 · framing

What's a meta-agent?

step=1  read app.py
step=2  edit app.py: add @app.route('/healthz')
step=3  pytest tests/test_health.py
step=4  pytest exit_code=1; ImportError
step=5  pytest exit_code=1; ImportError
step=6  pytest exit_code=1; ImportError
step=7  pytest exit_code=1; ImportError
▸ meta should halt here
step=8  pytest exit_code=1; ImportError

01 · what we measure

Eight capabilities.

Three families, each testing a different kind of meta-action: in-flight control, composition of tasks and traces, and safety & reversibility. Verification is cross-cutting and treated as a primitive, not a ninth capability.

← overviewcapability deep-dive · 1 / 8

@task
async def supervise(scope: Scope, issue: Issue) -> Patch:
    forks   = [scope.fork() for _ in range(K)]
    results = await asyncio.gather(*(fix(f, issue) for f in forks))
    best_f, best_r = select_best(
        (f, r) for f, r in zip(forks, results) if r.tests_pass
    )
    scope.merge(best_f); return best_r.patch

← overviewcapability deep-dive · 2 / 8

@task
async def supervise(scope: Scope, issue: Issue) -> Result:
    child = scope.fork()
    async for event in child.effects.stream():
        if shows_risk(event):
            child.discard()
            return await fix(scope.fork(), issue, hint=event)
    return scope.merge(child)

← overviewcapability deep-dive · 3 / 8

@task
async def orchestrate(scope: Scope, big: BigTask) -> Artifact:
    parts  = decompose(big)
    forks  = [scope.fork() for _ in parts]
    pieces = await asyncio.gather(*(do(f, p) for f, p in zip(forks, parts)))
    return integrate(pieces)

← overviewcapability deep-dive · 4 / 8

@task
async def optimize(scope: Scope, worker: Task, train: list[Input]) -> Task:
    runs     = [await worker(scope.fork(), x) for x in train]
    feedback = evaluate(runs)
    return propose_new(worker, feedback)

← overviewcapability deep-dive · 5 / 8

@task
async def supervise(scope: Scope, issue: Issue) -> Patch:
    child  = scope.fork()
    result = await fix(child, issue)
    if not result.safe:
        child.discard()
        return await fix(scope.fork(), issue, hint=result.cause)
    scope.merge(child); return result.patch

← overviewcapability deep-dive · 6 / 8

@task
async def supervise(scope: Scope, task: Task, budget: USD) -> Result:
    tier   = route(task, budget)        # cheap | strong | parallel | human
    child  = scope.fork()
    result = await tier(child, task)
    scope.merge(child); return result

← overviewcapability deep-dive · 7 / 8

@task
async def replay_distilled(scope: Scope, traces: list[Trace], new: Task) -> Result:
    workflow = induce(traces)           # offline distillation over past runs
    return await run_workflow(scope.fork(), workflow, on=new)

← overviewcapability deep-dive · 8 / 8

@task
async def supervise(scope: Scope, task: Task) -> Result:
    child = scope.fork()
    for event in child.effects.query(ToolOutput):
        if is_injection(event):
            child.discard()
            return Result.blocked(event)
    return scope.merge(child)

v1-lite retrospective · why we changed course

Per-capability benchmarks didn't work.

We tried six single-mechanism benchmarks, each scoring one meta-decision on synthetic data. Every one of them failed in one of four ways: saturated at the ceiling, inverted between models, flat (meta-agent equals baseline), or too hard (everyone at zero).

Diagnosis: hand-crafted single-mechanism data either ceilings frontier classifiers, fails to differentiate them, or hits a scaffold floor we cannot push through with prompt tweaks. The meta-agent is being measured as a one-shot LLM judgment, not as a multi-step operator over a real worker's coupled execution.

v2 thesis · today's benchmarks restart, ours replay

Our advantage: States-as-artifacts.

SWE-Bench and Terminal-Bench evaluate every run from raw environment state. Each evaluation is a fresh process, fresh disk, fresh worker stochasticity. Worker variance dominates the metric. MA-Bench freezes a worker trajectory once, with coupled agent and environment state, and lets every meta-agent operate on the same fixture.

v2 proposal · six states-as-artifacts tasks

Six states-as-artifacts tasks.

Each task is a real domain with verifiable outcomes. Same protocol, different capabilities exercised. The contribution is not "we invented six new task types" but "we standardize how meta-agents are evaluated over coupled agent and environment states on each."

← six taskstask 1 · meta-optimization · 1 / 6

Offline trajectory meta-optimization.

Read a pool of frozen worker traces. Diagnose recurring failure modes. Emit an improved worker. Tests whether a model can read execution evidence and edit policy.

← six taskstask 2 · multi-agent coordination · 2 / 6

N=3 / N=5 multi-agent coordination.

Extend CooperBench beyond pairs. N parallel sub-agents share one workspace; conflicts grow combinatorially. Tests whether meta-supervision scales with N.

← six taskstask 3 · safety forward-replay · 3 / 6

Safety with forward-replay.

Pre-recorded adversarial trajectories replayed step by step. Meta-agent must catch and revert harm before it crystallizes. Measures time-to-detection.

← six taskstask 4 · workflow extraction · 4 / 6

Repeatable workflow extraction.

5 example file migrations as training, 50+ held-out files. Meta-agent distills the rule. Applies it via replay. Verifiable by test-pass plus diff-semantic-equivalence.

← six taskstask 5 · budget routing · 5 / 6

Adaptive budget allocation.

Mixed-difficulty queue under a hard dollar cap. Difficulty labels hidden. Meta-agent routes per task to maximize aggregate pass_rate vs all-cheap, all-strong-truncated, and oracle baselines.

← six taskstask 6 · credit assignment · 6 / 6

Sample credit assignment for agentic RL.

Pre-recorded rollouts plus one-time offline ground-truth per-step advantage from K=8 counterfactual continuations. Meta-agent ranks decisive steps. Decouples fork-judgment from RL retraining.

02 · the contribution

One scaffold. Three primitives.

Underneath every direction is the same loop: worker acts on environment, meta-agent watches and decides. The layer couples agent state and environment state, so all decisions become composable.

Everything composes from these three. Plus inspect + intervene for the meta-agent's read/write side: Observe-and-intervene = inspect every step + intervene. Branch-and-select = fork(K) + score + revert(losers). Eight directions, five operations.

02 · scaffold diagram

Shepherd over sheep.

02 · substrate

Plug in any sandbox, any worker.

The state primitives live inside a sandbox. The worker is some agent harness. MA-Bench is the layer between them. Swap either side; the benchmark contract is unchanged.

substrate

Sandboxes

Modalfork · revert

DaytonaFUSE overlay

E2Bcode interpreter

Kubernetesself-hosted

worker harness

Sub-agents

Claude Codetools + skills

Codex CLIOpenAI agent

OpenHandsbrowser + shell

OpenCodeopen-source IDE agent

fork · replay · revert  ·  MA-Bench scaffolding layer  ·  inspect · intervene · trace

03 · the novel mechanism

Record once, replay many.

Because the scaffolding layer snapshots both agent state and env state at every step, a worker rollout recorded once becomes a portable fixture. Every meta-decision question over that trajectory is now a deterministic test.

03 · variance reduction

Worker variance cancels.

Live agentic benchmarks re-run the worker on every evaluation, mixing worker stochasticity into the meta-decision metric. Two seeds, two stories. With frozen fixtures, only the meta-agent moves.

Closest prior work: AgentRR records traces to guide the agent's future behavior. We do the opposite: freeze a trace, vary the meta-agent.

03 · variance PoC

What we measure to back this up.

For one direction (Observe-and-intervene), we run the same meta-decision N=20 times two ways: live re-rollout each time vs frozen-trajectory replay each time. The frozen path should collapse worker variance to near zero, leaving only the meta-agent's own stochasticity.

PoC plan: 20 trials × 2 conditions × 1 direction. Report mean ± std and bootstrap CI. Numbers above are illustrative; the experiment is queued.

04 · benchmark criteria

What good looks like.

Three bars. A flat leaderboard means the bench is broken; ceiling means stronger models have nowhere to go; toy data means frontier models have memorized the trick.

04 · what we borrow

What we borrow.

Each one clears a different bar. MA-Bench composes their moves: Meta Gain from CL-Bench, the orthogonal eval-grid axis from RE-Bench, and domain-uncorrelation from Hyperagents (V2).

05 · where MA-Bench fits

Where MA-Bench fits.

Production harnesses give workers tools. Multi-agent frameworks compose. Eval harnesses score the worker. Nobody standardizes the supervisor's decisions.

MA-Bench doesn't replace any tier. It consumes a production worker harness as substrate, exposes the three state primitives, and ships the 8 directions on top.

06 · honest status

Where we are.

Scaffolding layer works. Datasets are still hardening. The most important finding so far is reframing V2.

07 · for the room

Open questions.

Places where 10 minutes of discussion changes the next two weeks of work.

Appendix: site-format version · archived 18-axis taxonomy · V1 hardening pass 1 findings