A sandbox the agents are allowed to break

Give the agents a place where a failure costs an afternoon. On Node0 that place is two old laptops, sandbox1 and sandbox2, on a shelf in rack 3, running a virtualisation lab role and nothing the site depends on. An agent that is about to operate something for the first time builds it there first, breaks it on purpose, writes down what it found, and only then goes near the real hardware.

Three designs went through the lab before they went live, and each one came out changed.

The Ceph cluster was built as three virtual nodes on 20260816. The agent placed failure domains, replaced a disk, upgraded the cluster, and then killed all three nodes at once; quorum reformed on its own in about 90 seconds. The runbook that came out of it marks every finding as verified in the lab or as needing hardware, and the production cluster, nine servers and 170 OSDs, was bootstrapped from bare metal in a little over two hours on 20260903 because the design was already known to work.

The firewall pair was built as virtual machines on 20260817. The lab is where the firewall’s resistance to being managed like a NixOS host was worked through: configuration edits that fail to hot-reload without saying so, an API that never syncs a change to the peer, a DHCP failover that took four separate defects to get right, and the finding that a failover between the pair costs nothing measurable. When the real boxes arrived on a newer release, two of the lab’s findings came out different, which is the second half of the rule: the lab proves the design, and the hardware re-verifies the lab.

The remote-access redesign, in the week of 20260905, assumed that a firewall failover would carry the tunnel’s connections across because the shared address moves. A lab test built to check exactly that, with a forced one-sided failover, proved it false. The shipped design routes per firewall instead. Lesson 4.13 carries the outage that started it; the point here is where the wrong assumption died.

The hypervisor cluster went the same way. The whole Proxmox and Ceph deployment was modelled as virtual machines on sandbox2, a 2012-era business laptop, before any real hardware was tested, and a machine that old had the capacity for it with room to spare; the cluster keeps two throwaway guests of its own so that a guest image path can be tried before a real guest depends on it. The same habit runs at smaller scales. A consumer access point with a default DHCP server was staged on a spare network port of a sandbox, so that its rogue leases could never reach the live network.

What transfers is not the laptops. It is the rule that an agent’s first attempt at anything is made where the blast radius is a shelf, that the findings are written down with what the lab could and could not prove, and that the real hardware is treated as a second experiment rather than a deployment. On a site with one of everything, the sandbox is the only place a failure is cheap.