An alert redesigned by hand three times before it survived independent review

The original alert said one thing and only one thing: the cluster has been in a warning state for 15 minutes. It said that whether one disk was slow or three were failing. A second problem arriving on top of the first produced no visible change at all, so the alert quietly turned into a standing reminder that no longer described what was wrong. Every person who read it had to go and find out for themselves what it meant, every time, for the life of the system.

The rewrite, one alert per specific check and cause, naming the condition and the affected component, went through three independent reviews and each one found a genuine bug. The first version could never show a warning escalating to an error, because the metric it used locked in whatever severity a check was first observed at and never updated it. The second still leaked raw diagnostic text into a field intended to carry only a safe summary. It could also let an existing “this is expected” mute expire and then page for a condition that had already been accepted as known.

Both of those bugs are the kind that surface months later under a specific unlucky sequence of events, which is exactly why review found them and normal testing would not have. The result is now covered by a test suite whose cases exist one-to-one with the defects review caught.

The general point is that an opaque alert is a debugging cost paid every time it fires, forever, until someone redesigns it once. And naming an alert precisely is not cosmetic work. It is where most of the remaining subtle bugs in an alerting system tend to be hiding.