A capacity alarm that compared totals paged every night, and a notifier that did not log its own failures went silently blind

Two mistakes in one script arrived at the same destination, an alarm system nobody could trust, from opposite directions.

The first was a capacity guard that compared the total size of the dataset it intended to copy against the free space on the destination. That comparison is correct on the first night and meaningless on every night after, because a mature backup rotation works precisely by having nearly everything already present at the destination, which is also why little space is free. So the guard fired every single night, for weeks, while the actual work each night was fifteen seconds of new data. Any alarm comparing one mostly-unchanging total against another mostly-unchanging total will settle into a permanent state and stop carrying information. The fix is almost always to alert on the marginal change rather than the total: what is new tonight, against what is free tonight.

The second is the more important one. The script’s own notification helper caught and discarded its own delivery failures. When the notification service was unreachable for a period, a capacity alert and a health alert were both lost, with no record anywhere except a debug line nobody read. It later turned out that a nearly-truncated token file had been quietly failing every notification attempt. A component whose entire job is telling you something else has failed cannot, by construction, be relied on to alert about its own failure through its own channel. It must therefore log that failure somewhere durable, so that a broken notifier at least leaves a trail for whoever eventually goes looking.