What an offsite drive protects, and the guard that let it overflow anyway
- date
- 20260818
- what happened
- A nightly offsite USB backup failed because the drive lacked 0.1 TB of free space. The capacity guard detected it, logged it and sent an alert, then the script proceeded anyway, because the code path had no exit statement after the warning. A deeper audit found that roughly 22 TB of irreplaceable data had exactly one offsite copy, this drive, while several terabytes of already-cloud-backed media consumed drive space redundantly.
- what it cost
- Nine datasets corrupted on the offsite drive by the overfull run, and an inverted risk posture in which the best-protected data was consuming space the least-protected data needed.
- what changed
- The missing exit was added, so a predicted capacity failure now stops the run. Christoph made explicit written decisions about what to cull first and what to keep no matter what. A larger drive and adding the sole-copy archive to Backblaze were both costed and explicitly declined for the time being.
- the check now
- The capacity guard's failure path is a hard stop, verified by code review rather than by waiting to watch it fail again. The what-is-protected-where table, dataset by dataset, is a maintained document rather than an assumption.
A capacity check that correctly identifies a coming failure is only useful if something actually stops when it fires. Here the check logged the right number, sent the right alert, and then execution fell through into the very operation it had just warned against. The code path that logs a warning and the code path that halts execution were two different things, and only one of them existed. The immediate fix was one missing line.
The more interesting finding came from investigating why the drive was full at all. A protection audit revealed that irreplaceable, sole-copy data and thoroughly redundant data, already safe in three separate places, were competing for the same limited offsite space, with no policy deciding which one wins when it runs out. Fixing the software bug without fixing that policy question would only have delayed the next overflow.
The lasting change was making the tradeoff explicit. Someone looked at what each terabyte was actually protecting, decided in writing what to sacrifice first, and declined both obvious permanent fixes, a bigger drive or a bigger cloud budget, for reasons recorded rather than silently deferred. A guard is not finished when it detects a problem. It is finished when it stops the thing it detected, and that is a property you should read in the code rather than wait to observe.
Source: node0 lessons v0.1, lesson 3.9. Sanitized: checklist v0.1, 20260921; dataset names, cost figures, drive location; voice pass 20260921. Part of oznog.com/node0.