Creating an inventory record for a new host starts paging about it before it is built
- date
- 20260915
- what happened
- On 20260915 a record for a new wall-mounted display device was created in the fleet inventory system with its monitoring class set. Inventory records with a monitoring class become live scrape targets within a minute or two, so the fleet's critical host-is-down alert began paging eleven minutes before the monitoring agent on the new device was installed and running.
- what it cost
- An eleven-minute false critical page for a build that was proceeding entirely normally.
- what changed
- The standing onboarding practice for any new host: either take a scoped, time-limited silence for that specific device before creating its inventory record, which is the common case because the record is often how the build is tracked, or install the monitoring agent first and create the record afterwards.
- the check now
- Any host-down page during a known build window is treated as requiring verification rather than dismissal, because the hazard of this pattern is precisely that a real failure during a build looks identical to expected build noise.
In this fleet, the inventory system is not documentation. NetBox drives service discovery. Writing a device record with a monitoring class assigned is therefore the same action as arming that device’s alarm, and it takes effect within a minute or two. Creating the record for a new wall-mounted display was a purely administrative act in the mind of whoever typed it. Eleven minutes later, before the monitoring agent that would answer the scrape had even been installed, the critical host-down alert was paging about a build that was going perfectly well.
This is the same underlying mistake as shipping an alert before its producer exists, arriving through a different door. There, alerting configuration was deployed ahead of its data source. Here, a record in a system of record had an immediate operational side effect that nobody writing the record was thinking about at the time.
Any environment where infrastructure-as-data drives automatic discovery has this hazard built into its design, and that coupling is usually a feature. It is why new hosts get monitored without anyone remembering to add them. The fix is therefore procedural, not technical. Build the habit of asking, before writing any record into a system known to trigger automation, what that write is about to cause, and pre-empt it with a scoped, time-limited silence rather than discovering the page afterwards.
The check that came out of it is the uncomfortable half. Because expected build noise and a real failure during a build produce identical pages, every host-down page during a build window has to be verified rather than waved away.
Source: node0 lessons v0.1, lesson 2.14. Sanitized: checklist v0.1, 20260921; device name, field names; voice pass 20260921. Part of oznog.com/node0.
