Building a dead-man's switch that only proves itself by being killed

The design rule is one sentence: an alarm that travels through the machine it watches is not an alarm. The day before the founding brief was written, the primary storage server had been dark for hours with nothing said. So the notification server was deliberately put on a different host, and the alerting engine was set to post a permanent heartbeat alert to it every minute. If that heartbeat stops arriving for about ten minutes, the notification host pages on its own, without needing anything from the fleet’s metrics stack.

Then the team killed it to see. Stopping the metrics engine and timing the page produced a working alarm, but at 13 minutes 5 seconds rather than the intended 10. A delayed-resend setting on the alerting side had kept the heartbeat’s last known value alive for three minutes after it had actually stopped. That number is 30 per cent over target and only a live test would ever have produced it; reading the configuration would not have.

The second piece is for anything that can push data out but can never be reached to be asked how it is. A canary device sitting behind a one-way network boundary can report a problem, but it can never report its own death, because death looks exactly like having nothing to say. Anything in that position needs its own separate heartbeat watched from the other side of the boundary.

Two things to take away. Put the watcher of the watcher on different hardware and a different power feed, or the alarm and the failure can die together. And treat an untested dead-man’s switch as a design document until it has proved itself by being killed.