The NAS boot flash failed, and the backup built to survive it
- date
- 20260808
- what happened
- An internal USB2 port on the NAS motherboard corrupted the boot flash drive that holds the entire Unraid operating system configuration. Recovery succeeded only because a backup from months earlier happened to exist, written by an older ad hoc script that put unencrypted archives back onto the same box it was protecting.
- what it cost
- A near-miss of full configuration loss with no fallback, averted by luck. The old archives also sat unencrypted on the box while containing host keys and account data.
- what changed
- A nightly encrypted flash-backup script with two destinations off the box entirely: a Syncthing-replicated copy reaching four laptops, and a copy inside the existing offsite USB drive rotation. AES-256, because the archive contains host keys and replicates to four separate machines.
- the check now
- The script writes only when the configuration fingerprint changes, and runs at 23:30 rather than midnight so it lands inside the same nightly snapshot the offsite rotation carries that night. A monitoring rule tracks that it actually ran.
An appliance NAS keeps its entire personality, network configuration, container definitions, user accounts and host keys, on a small internal USB flash drive separate from the data disks. That flash drive failed. The only reason this was recoverable at all was luck. An old backup happened to still exist from months earlier, written by a script nobody had thought carefully about.
That old script had a design flaw invisible until the day it mattered. It wrote its backup onto the very box it was protecting, so if that box’s boot media dies, the backup dies with it. The replacement design is a short checklist for backing up any single-purpose appliance’s own identity: encrypt it, because configuration backups tend to contain credentials by necessity, and put every copy somewhere that survives the box itself being dead, not just somewhere convenient.
A small scheduling detail mattered too. The backup is timed to land inside the same nightly snapshot cycle that the downstream offsite process consumes, rather than at a round clock time that happens to run just after it. That cost nothing and closed a one-day propagation gap that would otherwise have repeated forever. Writing only on a fingerprint change keeps the history readable. A new archive means the box’s configuration actually changed that day, which is itself a useful signal.
Source: node0 lessons v0.1, lesson 3.8. Sanitized: checklist v0.1, 20260921; script, rule and credential filenames; voice pass 20260921. Part of oznog.com/node0.