Skip to content
Oznog

3.12 · storage and backups · as-is

A backup nobody has restored is a belief, and skipped is not passed

date
20260821
what happened
A weekly automated restore rehearsal went live. For a rotating sample of repositories it opens each one with its stored password, restores real files from the newest snapshot to a scratch directory, byte-compares the restored size against what the snapshot recorded, then deletes the scratch copy.
what it cost
Nothing as an incident. The cost it closes is the invisible one: an unproven backup everyone believes works until the day it does not.
what changed
A dedicated restore-rehearsal service, run weekly against three repositories per run out of a fleet in the tens, so each repository is exercised roughly every six weeks. It reports three outcomes, not two: passed, failed, or skipped.
the check now
Skipped repositories, those that could not be opened or had no file in the sample size range, raise their own alert rather than being counted as fine. The rotation cursor is stored as a repository name, not a numeric index.

Backup verification tends to stop one step short of the thing anyone actually cares about. It is easy to confirm a snapshot was taken, and not much harder to confirm the backup’s internal bookkeeping is structurally sound. Neither proves that a real file, fetched over the network, decrypted and reassembled, comes back out intact. This system was built to close that last gap: an actual small-scale restore every week, compared against what was recorded, rotating through the whole fleet’s repositories over time rather than testing the same one repeatedly.

Two design choices make it trustworthy rather than merely present. First, the rotation position is tracked by the name of a repository, not its position in a list, because a numeric position silently means something different the moment the fleet grows or shrinks. That quietly reduces what actually gets tested while the dashboard looks unchanged. Second, and more important, when a repository cannot be tested for any reason, the system reports that explicitly as “skipped” and alerts on it, refusing to let an untestable repository disappear into a clean-looking summary.

A repository that is merely unproven is treated as exactly that, not quietly counted alongside the ones that passed. Any verification system with a pass/fail shape should be checked for the same gap: what does it do with the cases it could not evaluate, and can you tell those apart from successes at a glance.

Source: node0 lessons v0.1, lesson 3.12. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.