A folder permission left from a partial deploy silenced the entire alerting path for 98 minutes

A deploy script wrote a shared secrets folder with a restrictive mode and intended to relax it thirty lines later in the same run. Anything that read the folder in the window between those two lines would fail. The alerting engine’s own container did, because it needed a credential from there to deliver anything at all. For 98 minutes the fleet had no alerting. Not degraded alerting, none. No heartbeat, no warning, no critical page.

The part that makes this the load-bearing proof of the dead-man’s-switch design is what could not catch it. The alerting engine has its own internal self-check, an alert whose job is to notice when notifications are failing to send. It could not fire, because the very credential it needed to send that alert was the thing that had just been locked away. An in-band self-check cannot report a failure that takes out the band.

What did catch it was the second, structurally independent watcher: a different machine, with its own delivery path, watching for the heartbeat that arrives every minute and paging when it stops. It had no opinion about permissions or containers or deploy scripts. It only knew that the fleet had gone quiet, which was true.

The generalisable rule is that a system’s alerting path needs a check that does not depend on the alerting path working. The second rule, smaller but cheaper to act on, is that a deploy script which tightens a permission and then loosens it has created a window, and anything reading during that window, including a person doing the same task by hand, hits it.