Silencing a host by name during maintenance muted almost nothing the maintenance actually caused
- date
- 20260917
- what happened
- A planned maintenance window on a Ceph storage host on 20260917 was preceded by a monitoring silence scoped to that host's own name. Powering the host down paged eight separate alerts anyway, because several rules that fire during a host outage name the cluster, or the switch, as their subject rather than the individual host.
- what it cost
- Eight unnecessary pages during a planned, expected maintenance window, despite the documented procedure having been followed correctly in every other respect.
- what changed
- A dedicated maintenance-silence tool that silences by every label under which a host can appear as the true subject of an alert, so one command scopes a silence correctly regardless of which rule's convention applies.
- the check now
- The tool's own self-test validates its label matchers against recorded label sets from real alerts, and specifically guards against a silence for one host's name matching a different host whose name happens to start the same way.
Scoping a maintenance silence to the name of the host you are about to power down is the obvious thing to do. It is what the existing runbook said, and it had worked once before. On 20260917 it silenced almost nothing that the maintenance actually produced. Eight alerts paged.
The reason is structural rather than a mistake anyone made on the day. Alerts exist at more than one layer of aggregation, and the label that says who an alert is about reflects the layer the rule lives at. A rule watching the storage cluster names the cluster as its subject, because from the cluster’s point of view the affected entity is the cluster. A rule watching the network layer names the switch, because the switch is the thing reporting; the host that went down appears only as a neighbour reference buried in the alert’s detail. Neither of those is wrong. Neither of them matches a silence keyed to the host’s device name.
There was no missing tool and no missing procedure here, just a mismatch between how a person naturally thinks about scoping a silence, which host is this, and how a monitoring system assigns the label meant to answer that question. The fix is a maintenance-silence tool that has already enumerated every label shape an alert about a given host can carry, and silences on all of them from one command.
Any monitoring system with alerts at more than one layer has this waiting. Before the next planned outage, it is worth taking the label sets from a real past outage of one host and checking how many of them actually carry that host’s name.
Source: node0 lessons v0.1, lesson 2.16. Sanitized: checklist v0.1, 20260921; tool name, naming conventions; voice pass 20260921. Part of oznog.com/node0.