An alert fired continuously on the normal state of a design it misunderstood, and was removed rather than tuned

A monitoring rule can read its target correctly every single time and still misrepresent the system. This one did. The fleet advertises internal routes over a mesh VPN from two redundant routers, and the metric the alert watched reports which of them is currently the elected primary. In a healthy, correctly redundant pair, exactly one router is not the primary, at all times, forever. The alert had been written by someone whose mental model of “advertising the route” did not match how the failover software actually behaves when there are two of them. So it reported the design working as intended as a fault, continuously, from the day the second router was added.

The instinct when an alert misfires is to raise its threshold or add a delay. No threshold fixes an alert built on a wrong premise; it only makes the wrong alert quieter. The rule was deleted.

The useful part is what was found in its place. Reading the broken alert’s own description text showed it had been trying, badly, to say something real. Nothing anywhere checked whether the standby router was still an approved participant in the route at all. An unapproved standby looks completely fine until the moment a failover needs it, which is the worst possible moment to discover it. That became the replacement alert, along with a flapping check on repeated primary elections, and a guard so that the router-management service being down reports as itself rather than as a loss of redundancy.