Verifying an agent self-improvement loop design against the code it assumes

An LLM drafted me a polished technical design doc: an autonomous evolve loop where a swarm of coding agents systematically improves our product agents. Rank N agent surfaces by eval deficit, fan out M hypothesis-implementing pods per surface, eval each candidate, merge the winner, repeat. Tickets as the distributed mutex, PR bodies as the metadata store, everything stateless and crash-recoverable.

It read complete. Then I verified every load-bearing claim against the three codebases it touches — the swarm runtime, the eval service, and the MCP gateway — and three of its core assumptions died on contact with source.

What died

1. The eval can't see candidate code at all. The design's whole premise is “implement hypothesis on a branch → launch eval → compare against baseline.” The eval service's actual tool surface has no git ref, no sha, no branch parameter anywhere in the launch chain — it runs a checkout baked into the worker image. Every launch evaluates main-as-of-image-build; baseline and candidate carry identical provenance, and the comparison tool would correctly report zero variables changed. The candidate's diff is structurally invisible. The fix is one parameter on the eval-submit flow — but nothing else in the design matters until it lands. The cheap parts (tickets, labels, routines) are all buildable today, and building them first would just be concluding around the missing experiment.

2. The dispatch layer enforces one run per ticket. The design wants M parallel implementors on one ticket — first wins, the rest bounce off the admission gate. Workaround costing zero code: the ticket field is a free-form string used only by the gate, so TICKET::m1..mM gives per-candidate dedup. Verified by code read, not yet live-probed — flagged as hypothesis.

3. Autonomous merge doesn't exist here. Org ruleset requires human approval; the bot can't self-approve. Resolution was a scope cut, not code: the loop labels the winner, humans merge — which matches actual 2026 practice, where documented cases of reviewless autonomous merging to production are essentially absent.

What survived, amended

• Two routines collapsed into one idempotent reconcile tick. Every action is recorded where it happened; any tick can crash and re-derive the walk.
• Evals launch from the routine, not the pods. The pods' tool surface stays hermetic; the routine reads the PR head sha after delivery. No eval spend on runs that never delivered.
• Losers need explicit cancellation. Closing a losing PR doesn't terminate its run — it parks waiting for CI feedback forever. Close + cancel, always.
• Stacking works until the merge. Rounds can stack on the unmerged winner branch; squash-merge at the stack's bottom means children can't rebase cleanly. Cap stack depth, treat human merge as the flush point.

The comparison that reframed it

Grounding against current practice produced a bigger amendment than any single bug: prompt-class improvements shouldn't use coding-agent fan-out at all. The reflective-optimizer family (GEPA et al.) runs candidate prompts in-process against the metric — roughly two orders of magnitude cheaper than a full pod + eval suite per candidate, and the entire mutex/fan-out apparatus evaporates for that lane. The evolutionary-code-search family taught the inverse lesson: every deployed success story ran on cheap deterministic evaluators; ours are slow LLM-judge suites, so sample efficiency is the scaling axis, not parallelism. Final shape: optimizer job for prompt-class strategies, agent-PR loop only for code-class strategies, human merge as the gate on both.

Coda: it's a graph either way

Mid-2026 discourse calls this “graph engineering” — loops wired into directed graphs with routing and state. This design is that, with one property the framework versions get as a feature and this one gets structurally: the graph state lives in external durable systems, so the controller is stateless and any tick resumes the walk. The loop's endgame is the 2026 frontier item: a graph that rewrites its own nodes — eventually its own topology.