A monitoring tool with zero monitors configured

After a week spent building infrastructure monitoring quickly and under pressure, the team stopped and audited what was actually covered. The answer was uncomfortable. Hosts, disks, switches and the storage cluster were well watched, and the services those machines exist to run were almost entirely unwatched. Only a handful of the running application containers had any check at all. Worse, an uptime-monitoring tool deployed some time earlier, which from the outside looked exactly like service coverage, turned out to have no monitors configured inside it. It was running, healthy, and checking nothing.

A tool that looks like it is doing a job and is not is more dangerous than no tool, because it changes what people believe they can rely on. The zero-monitor case is the sharpest example here, but the broader finding is the one that generalises. Building monitoring for the infrastructure a service runs on tells you nothing about whether the service is doing its job. It is entirely possible to spend weeks building solid infrastructure monitoring while the actual product is silently broken the whole time.

The fix was a catalogue with three states for every service: answered by its own application-level question, probed with a body check by decision, or port-only and absent with the reason recorded. The surviving discipline is never “is the port open” or “is the process running”, always “ask the service its own question and check the answer”.

The same pass produced a second rule worth lifting out on its own. For anything that accumulates a lifetime failure count, alert on the change over a fixed window, never on the running total, because a cumulative counter never returns to zero and an alert on its raw value pages forever until someone silences it and stops reading the next one too.