Editing the config by hand took out both nightly backups at once
- date
- 20260909
- what happened
- A single atomic snapshot of all data is taken every night before either of two backup plans, one to Backblaze and one to an object-storage bucket on the site's own Ceph cluster, is allowed to read anything, and a generator script owns and validates that configuration. Someone moved one of the two plans onto the consistent snapshot by editing the running configuration file directly instead of through the generator.
- what it cost
- One night with zero backups running to either offsite destination. Recovery took about twenty minutes, because the abort behaviour was itself correct and left the previous night's snapshot view untouched rather than stranding it half-built.
- what changed
- Nothing about the safety mechanism, which worked as designed. What changed was the written instruction that hand-editing the configuration file is never the way to make a change, plus a documented procedure: edit the generator's source spec, run it to validate, run it again to write, then restart the daemon.
- the check now
- The generator's check mode must report clean before any change is considered complete. A hand edit to the live configuration file is treated as an incident to recover from, not as a valid way to make a change.
A system built with real safety mechanisms can still produce a bad night. The mechanism did not fail. Someone worked around the process that keeps the mechanism informed. Two backup plans, writing to two entirely separate destinations, looked independent from the outside. Both actually depended on one shared step earlier in the night: a single atomic snapshot of all the data, taken once and read by both, specifically so that neither backup captures data mid-write.
The configuration file that normally only changes through a generator, which validates every change against a strict specification before writing it, was edited directly by hand to move one plan onto that shared snapshot. The validator caught the resulting inconsistency immediately and did exactly what it was built to do: refuse to proceed rather than build a backup on top of a configuration it could not trust. Both plans failed together that night, since both needed the step the validator had blocked.
In a real sense this was the system working rather than failing, and the fast, clean recovery bore that out. It is also a sharp illustration of blast radius in shared infrastructure. A change to one part of a system that appears self-contained can take down everything sharing an invisible dependency underneath it, and the fix is not to weaken the validator that caught it. It is to never again make that class of change outside the process built to check it.
Source: node0 lessons v0.1, lesson 3.17. Sanitized: checklist v0.1, 20260921; plan, bucket and script names; voice pass 20260921. Part of oznog.com/node0.