Runner Fleet Atlas

One sentence: the vendor queues a session; your own always-on poller notices and launches a one-session VM from a pre-baked image, which registers with a single-use token, serves the session, then powers itself off into termination.

Nothing pushes into your network. The queue lives at the vendor; the only thing crossing the boundary is an outbound poll. Reference notes from operating one such fleet.

Components

Environments

• Same image, same code — the personalities differ by one number.
• Disabled clocks are written out explicitly, even where the module defaults them to zero: an inherited value is invisible at the call site, so nobody auditing the block against its own promise can see a knob that isn't there.
• Why each environment costs a permanently idle machine: the picker only lists environments with a runner checked in, and a dispatch to an environment showing no capacity generates no hint — so zero standbys means the environment never appears and can never bootstrap.
• The standby also hides boot latency. The ~226s is still paid, just afterwards by the replacement, off the critical path.

What survives a resume

• The vendor stores a pointer, your git host stores the content. The session record holds a branch name; resume fetches that ref from your remote.
• Proof from the failure mode: delete a branch after its PR merges and resume dies with a missing-ref error. Re-creating the ref fixes it — which it wouldn't if the conversation carried the code.
• Automatable: merged PR heads stay reachable forever on the forge, so the branch can be recreated server-side at its final commit with one API call, no clone.
• Resume often lands on a different machine, or on an existing idle standby — which may still be running the old image.

Identities

Four controls keep the session below its host — each alone proved insufficient

Defence in depth fails when one layer hands out the thing the others protect.

The narrow-door pattern

• A session legitimately needs real app config; the tool that fetches it needs cloud credentials — the exact thing the firewall denies.
• Answer: one root broker per legitimate need, not a widened rule.
• Limits, all load-bearing: strict character allowlist on the argument, so a path can't escape the dev prefix and it can never return production; no passthrough flags, so it can't become an arbitrary invocation of the underlying tool; config only, never credentials.
• It pins its own cloud-config to a root-owned path — because that file is otherwise session-writable, and a credential-process entry in a config read by a root process is arbitrary root code execution.

The registration token

• Long-lived environment secret registers any runner and can claim any member's queued session — so it never reaches a host that runs session code. Orchestrator only, into a memory-backed filesystem.
• Executing host gets a single-use token, scoped to one session. Both go through the same CLI flag, which is the easy mistake.
• Staged in a parameter store, not instance user-data — user-data stays readable from inside the instance for its whole life.
• Encrypted at rest · written to tmpfs, never the snapshotted disk · deleted the moment it's read, so a later reader can't replay it · never logged.
• Lifetime matches the boot budget, not the session — a leaked staged token is inert within minutes.

Why runners cull themselves

• The permissions ceiling grants only the management agent's own registration calls. Remote command execution is absent, and no policy edit can add it — the ceiling caps what any role may grant.
• What is permitted: describe and terminate instances. So each runner decides for itself.
• Better anyway: a runner reads its own metrics, so it's never wrong about whether it is busy. Inferring another host's state from a launch-time tag can't distinguish a spare from a machine serving a person.

Baked vs ad-hoc

Which dependencies are real, which are containerised

The line follows what the dependency is for: previews read the world as it is, so real services give a truer preview. Tests need isolation, so they get containers. The preview path never starts a database.

The toolkit

Each is a separate clean context. A reviewer sharing the author's context shares the author's blind spots.

Reaching a machine nothing can reach

• Zero ingress → no port to expose → previews go outbound through a tunnel on a reserved wildcard domain.
• Tunnelled HTTPS serves one port. There is no base URL with varying ports — every reachable service gets its own hostname.
• Static artifacts take a different path: a CDN-fronted bucket, deliberately separate from durable scratch, because a URL you hand someone shouldn't live in a bucket every session can overwrite.

The numbers

Footguns, each observed live

Control surface

Audit periodically whether every knob you believe is applyable actually reaches the running process. A variable that is declared, documented, and never read looks identical to one that works.

Checklist: preparing an agent-friendly environment

Do first — hours of work, prevents lost runs

State

• Conversation on the platform, code on the forge, nothing durable on the box.
• A durable scratch store with a stable prefix across resumes.
• A separate artifact store for openable URLs.
• Rescue uncommitted work on exit by discovery, not declaration — a declared-checkout list misses the repo cloned on a whim.

Image

• Bake repos, dependency installs, toolchain. A stale bake costs a fetch; a missing one costs a cold clone every session.
• Bake the gitignored half too — generated schemas, env stubs, build outputs. Absent from a fresh clone by definition, and usually the biggest per-session cost.
• Warm service images if any test tier uses containers.
• Never bake secrets. Fetch at boot into memory.
• Pin per environment, so one can move while another holds.

Credentials

• The credential that can claim any work never touches a host that runs agent code.
• Executing hosts get single-use tokens, deleted on read, lifetime matched to boot.
• Narrow brokers, not blanket privilege — one command per need, each validating its input.
• Any config a privileged process reads must be owned by that privilege.
• Never a secret in a command line or a dumped environment — both land in the agent's own transcript.

Clocks

• Know exactly what defers your idle clock. “Defers while busy” and “defers while waiting to be busy” are different promises, and self-paced loops live in the gap.
• Every no-timer environment needs a reclamation path. At minimum, alarm on instance age.
• Distinguish a spare from a machine serving a person before anything may reap.
• Check whether spawn latency exceeds your capacity-evaluation interval.

Verification

• Assert on traces, not status codes: span exists, nested correctly, outcome says success, side effect landed, and nothing else changed.
• Read the warning level. A healthy response can hide four of five internal attempts failing.
• Two independent surfaces for any number the agent quotes. A mismatch is the finding, never noise.
• Assert a positive count. A job that matched zero tests is green, fast, and proves nothing.

What generalises

• Poll, don't listen. An outbound queue needs no inbound path → literally zero ingress → every other control gets easier.
• Make the machine the unit of work. One session per instance, self-terminating. No shared state to leak, no drift to accumulate.
• Split the credential by lifetime. Claim-anything secrets on the dispatcher; single-use tokens on the executor.
• Put state where it survives. Then losing a machine is an event, not an incident.

What doesn't generalise: the isolation model. Per-uid firewall rules on the metadata endpoint, narrowed sudo brokers, self-termination as the exit path — all shaped by a session running directly on a dedicated host. Move it into shared-kernel containers and none transfer; you'd re-earn each property with a different mechanism, and the failure mode would be silent.

The documentation trap

• The most expensive defect here was not code. A skill file claimed a lifetime guarantee that had been changed underneath it, so agents kept promising a ceiling that no longer existed.
• Another described a startup timeout as applying only before initialisation. It re-arms on every attach — that misreading cost an environment days of silent churn.
• Documentation an agent reads is executable, in the sense that the agent acts on it.
• So: keep measured constants out of prose nobody re-verifies, state the mechanism rather than the number, and tell the reader how to check the claim against the running system.