Skip to content
Oznog

2.20 · monitoring and alerting · as-is

Backrest's own error API caps at 21 entries, and an operator chased a fix that had already worked

date
20260819
what happened
The backup tool's operations API reports at most 21 error entries per run in its summary list, a display limit rather than a true count. Over 180 runs on one host it consistently reported exactly 21 while the contents behind that number changed completely from run to run, which made adding exclusion rules for known-noisy files look entirely ineffective. Found 20260819.
what it cost
Time spent doubting a fix that was working, and a real risk of abandoning an effective exclusion strategy as worthless.
what changed
The practice of reading the tool's underlying database directly rather than trusting its summary API for anything resembling a total or a complete list. The true count for one run, checked that way, was 39 against the 21 reported.
the check now
Any count from this tool's API is treated as a display sample, not a total, before any conclusion is drawn from a change in it, or from the absence of one.

Backrest’s operations API returns at most 21 error entries per run. This is a documented display limit and not a bug. But the summary reads as a count, and a count is what anyone looking at it will act on.

The effect on the work was almost backwards from the usual monitoring failure. Excluding a known-noisy file from a backup run removed one error, which freed a slot, which another genuinely failing file promptly filled. The reported total stayed pinned at exactly 21 across roughly 180 runs. Every exclusion added looked like it had accomplished nothing, when in fact each one had worked exactly as intended. This was not a false alarm; it was a false absence of improvement, which is harder to notice because nothing about it feels wrong. Checking the true count for one run directly gave 39, nearly double what the API reported.

The fix was to stop asking the convenience layer and read the tool’s own internal database, a SQLite log it maintains, for anything that needs to be a total.

The habit worth generalising is a trigger rather than a rule. When a number stops responding to a change that should have affected it, suspect the number before concluding the change did nothing. A capped list masquerading as a complete answer does not fail loudly; it tells a partial truth that happens to look total, and it will keep doing so indefinitely. Any count arriving through a summary or convenience API, particularly a third-party one, deserves that suspicion the first time it refuses to move.

Source: node0 lessons v0.1, lesson 2.20. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.