The recursion rule: a backup of a backup is worse than no backup
- date
- 20260905
- what happened
- While rebuilding the backup topology around the new central server, Christoph set an explicit rule: a destination must never hold a backup that originated from itself. The Ceph cluster's own landing directory on the central server is excluded, by code, from the plan that backs the central server up into Ceph, and Ceph's own identity bundle is written only to Backblaze and to rotating USB drives, never back into Ceph.
- what it cost
- Nothing directly, since this was designed in rather than recovered from. It closes a failure mode that would otherwise only be discovered during an actual disaster.
- what changed
- The rule is enforced in code, not left to memory. The generator that builds the backup tool's configuration hard-codes the recursive exclusion for every plan writing to a given destination, and refuses to write a configuration that violates it.
- the check now
- The generator's own validation fails closed if a managed plan's excludes do not match its specification, which includes the recursion-rule exclusions. Nightly runs abort rather than silently produce a self-referential backup.
A multi-tier backup design, where several systems back each other up in different directions to spread risk across independent failure domains, has one obvious way to quietly fail. A loop, where system A’s backup of system B includes data that originated from A in the first place, gives the appearance of redundancy while providing none. It consumes real capacity and produces real confidence, and neither is worth anything.
The rule adopted here is simple to state and easy to violate by accident as a system grows, so it was written into the tool that generates backup configurations rather than trusted to human memory or to a document nobody rereads. Every plan the generator produces is checked against the rule before it is allowed to run, and a plan that would violate it, even through an indirect path introduced by some later change, fails to validate and the nightly run refuses to proceed.
The generalisable idea is not really about backups. Any system with multiple redundant copies feeding into and out of each other benefits from one invariant, stated in plain language and then enforced by code rather than by convention, that closes the specific way redundancy quietly becomes illusory. Writing the invariant down is the easy half. Making the pipeline refuse to run without it is the half that still works in two years.
Source: node0 lessons v0.1, lesson 3.14. Sanitized: checklist v0.1, 20260921; dataset, bucket and script names; voice pass 20260921. Part of oznog.com/node0.