A written-down cluster quorum threshold became wrong, in the dangerous direction, the day a fourth member joined

A number typed once into a monitoring rule, correct on the day it was written, became a landmine the moment the system it described changed shape. The rule paged when fewer than two members of the virtualisation cluster were answering, which is right for three members. A fourth member joined. Quorum for four is three, so a two-of-four cluster is genuinely frozen, and the rule would have said nothing at all. That is the dangerous direction. A rule that fails to fire when it should is more dangerous than one that fires too often, because nothing about it looks broken from the outside.

The fix is to compute the threshold from the live counted membership at evaluation time rather than writing it down. That generalises to any alert built on how many things exist, whether the things are cluster members, redundant paths, power feeds or hosts in a fleet. If the population can grow, the number cannot be hand-typed.

The same inspection found the same root problem wearing different clothes. Rules meant to detect a whole rack losing power at once were counting guest virtual machines as if they were independent physical hosts. So one hypervisor rebooting and taking its guests with it could cross a threshold that was supposed to mean several distinct physical machines had failed together. The rules were also lumping every device with no rack assignment into one nameless group and correlating across it. Both guards were proven load-bearing the honest way. Each was removed individually, and the test was confirmed to fail without it.