Four monitor bugs in one week, and the silent ones were the dangerous half

Writing monitors quickly, under pressure, during live incident response produced four broken monitors in a single week. The four are worth listing because they fail in two different ways. The first parsed a column out of a cluster status command and reported zero inactive placement groups while five were blocked and client I/O had stopped. The second was written as a shell variable holding a whole remote-login command string, which works under bash and, under the environment’s actual default shell, becomes a single nonexistent command name, so it failed on every run and checked nothing. The third pinned the log it watched to the current date, so at midnight a perfectly healthy 24-hour job triggered an alarm. The fourth was a firmware-event watcher whose own login to the management controller wrote an event-log entry, which the next poll dutifully reported as a new event, forever.

The pattern matters more than any of the four. A monitor that fails loudly, the wrong shell, an immediate crash, a nightly false alarm, is an inconvenience. You find out about it quickly because it annoys someone. A monitor that fails silently and keeps reporting a plausible “all clear” is a liability, because nothing distinguishes it from a genuinely healthy system until the failure it was meant to catch arrives and it misses that too.

The answer adopted here is a discipline rather than a tool. Do not trust a check until you have deliberately broken the thing it watches and confirmed it noticed. A parsed number that looks right is not proof. A caused failure that gets caught is.