Shipping an alert before the thing it watches exists pages a human about your own unfinished work

The rule was not buggy. It did exactly what it was designed to do. It detected exactly the condition it was written to detect: nothing was publishing the health-check data it expected. That was true. The collector that would publish that data had not been built yet, because it was still waiting on review, and the rule had been committed alongside it as part of the same piece of work.

The mistake was entirely one of sequencing. Configuration in the live rules directory is live the moment it lands there; there is no separate act of turning it on. So a piece of half-finished work became a production alert, waited until the middle of the night, and woke someone up to report on the unfinished half of itself.

The fix required no engineering at all, only a decision about where work-in-progress configuration is allowed to live. Anything not yet backed by a running producer stages outside the deployed path, and gets promoted after the producer is confirmed live and its data confirmed present in the monitoring system. Confirming that the code exists is not the same check; the question is whether the series is populating right now.

This is a small mistake and an easy one to repeat, particularly in any environment where alerting configuration and the systems it monitors are built by different people, by different agents, or on different schedules. The cheap durable fix is worth adopting before the first four in the morning false page rather than after it, because an alert is a promise that something is already being watched, and this one broke that promise on the first thing it ever said.