One unique password, and a backup the verifier could not see
- date
- 20260917 to 20260918
- what happened
- A new fleet host was built with a freshly generated, unique backup-repository password, the intuitively more secure choice. The next morning the fleet's automated backup-verification sweep flagged it. The sweep could open only the repositories it had been set up to open, so a repository with a credential it did not hold was excluded from verification entirely.
- what it cost
- An unknown window, at minimum overnight, in which a fleet host's backups ran with zero automated proof they were sound, invisible because nothing failed loudly and the dashboard showed green.
- what changed
- A written credential policy for backup repositories that keeps every repository inside the sweep's reach, and a procedure for bringing a new repository into it. The procedure names every tool that has to learn about the repository, because each keeps its own list, and updating one leaves the others blind while their dashboards still look fine.
- the check now
- After bringing any new host onto the fleet's backup system, run the verification sweep against it explicitly and require a clean result before the host's backup setup counts as complete.
Security intuition and operational reality pulled in opposite directions here, and operational reality won for a specific, well-reasoned cause. Giving a new machine its own unique backup-encryption password is, in isolation, the more cautious choice: no single compromised secret affects more than one host. But the fleet’s automated proof that backups are sound, a nightly and weekly sweep that opens every repository and checks it, can only open the repositories it holds credentials for.
A repository it cannot open, whether by oversight or by a well-intentioned security choice, is invisible to the sweep, and invisible in a particularly dangerous way. The backup itself keeps running, uninterrupted, so every visible signal says everything is fine. The one system meant to catch a broken or corrupted backup never looks at it.
The fix adopted was not a smarter sweep. It was a written policy for how backup repositories get their credentials, chosen so the sweep covers every one, and a procedure for bringing any new repository inside it. The principle generalises well past backups. A new component is not finished integrating into a fleet when its own function works. It is finished when every fleet-wide system that is supposed to be watching it can actually see it. That has to be checked explicitly rather than assumed from the component working in isolation.
It produced rule 10 .
Source: node0 lessons v0.1, lesson 3.19. Sanitized: checklist v0.1, 20260921; hostname, secret paths; voice pass 20260921; credential arrangement generalised 20260929. Part of oznog.com/node0.