The append-only backup server's orphan-pack loop
- date
- 20260727 to 20260803
- what happened
- A restic client failed repeatedly against an append-only backup server, and the diagnosis was blamed on server load, so the backup was paused for a day. The actual cause was a blob upload that timed out client-side but landed server-side; the retry was refused as a duplicate write, leaving orphaned data blocks with no index, so every subsequent run recomputed and re-uploaded the same blocks and died again.
- what it cost
- A day of a laptop going unbacked-up while the wrong cause was chased, and later, on a second host, a repository that looked empty while holding gigabytes of unindexed data.
- what changed
- A documented recovery sequence: confirm any stale lock's process is genuinely dead, run restic unlock (explicitly permitted even though the server is append-only), then restic repair index --read-all-packs before any retry. Load is no longer the first thing checked; orphan packs are.
- the check now
- Any interrupted backup to the append-only server runs unlock then repair index --read-all-packs before retrying, regardless of how busy the server looks. The failure signature "-1, no summary event found" is read as "the process was killed by a signal", not "the backup failed on its own".
Backup systems built to resist deletion, an append-only mode so that a compromised client cannot erase history, create a failure mode that looks nothing like what it is. A network hiccup during an upload left data blocks on the server with no index entry pointing at them, orphaned rather than committed. Because the server refuses to let a client overwrite anything, the retry could not simply replace the failed upload. It kept recomputing the identical blocks and getting refused every time, which looked exactly like an overloaded server rejecting work under pressure. A day went into chasing that theory before the retry succeeded at an even higher load, disproving it.
The real fix is a repair step that rebuilds the index from what is actually on disk, run reflexively after any interrupted backup rather than diagnosed fresh each time. A second occurrence added a stale lock left by a process a reboot had killed mid-run, which produces an entirely different-looking error, “already locked”, masking the same underlying cause.
The general shape is worth carrying. When a system is deliberately hardened against one kind of failure, here deletion, watch for the failure mode that hardening creates instead, here orphaned partial writes. Build the one-line repair into muscle memory rather than treating each occurrence as a fresh mystery.
Source: node0 lessons v0.1, lesson 3.5. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.