A hand-edited config that hides its own drift for weeks
- date
- 20260818
- what happened
- Backrest reads its configuration file only at process startup, never on a hand edit to the file on disk. A change adding a 35 GB directory to a laptop's exclude list took effect on disk but not in the running process, so the file said excluded while the running backup still included it.
- what it cost
- Nothing realised, because a proactive fleet-wide sweep caught the drift before the excluded data ever vanished from a backup. The exposure was open-ended: the change would have taken effect at whatever unpredictable future moment the process next restarted.
- what changed
- A standing rule: after any hand edit to the configuration file, restart the daemon, or make the change through its own interface or API instead, because the running process never re-reads the file on its own and can silently drift from what is on disk. Verify against the live running configuration via the API, never by reading the file on disk.
- the check now
- A disk-versus-live exclude-count comparison swept across the fleet, comparing the file's exclude count to what the running process reports through its local API. It found exactly one drifted host out of twelve on first use.
Many long-running daemons read their configuration once at startup and never again. That is an entirely reasonable design choice, and it becomes a trap the moment someone edits the file directly instead of going through the application’s own change mechanism. Here a hand edit correctly landed on disk, and the process kept running on its stale in-memory copy with no indication anywhere that the two had diverged.
The dangerous part is the timing. The edit does not simply fail to take effect; it becomes a change staged to activate at some unknown future moment, whichever routine event happens to restart the process next, an operating system update, a crash, a planned reboot. That turns an ordinary maintenance action into an unplanned configuration change days or weeks later, with no correlation an operator would think to check.
Reading the file and finding it correct proves nothing about current behaviour. The only trustworthy source is the running process itself, queried through whatever live interface it exposes. A fleet-wide sweep comparing what the file says against what the process reports turned out to be cheap enough to run as routine hygiene, and caught a real silent drift on its very first pass, one host in twelve.
Source: node0 lessons v0.1, lesson 3.10. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.