Monitoring and alerting on Oznog

The day four failures went unnoticed, and none of them paged anyone

Sun, 06 Sep 2026 00:00:00 +0000

Four separate failures were in progress at once on 20260906, and the fleet said nothing about any of them. Two Ceph hosts had suffered a CPU internal error and hung for ten hours with chassis power on, network links up and their operating system journals simply blank. They were found because both machines started pushing roughly six hundred Ethernet flow-control frames a second into the switch fabric, which someone noticed while looking for something else. Four switches were running their physical-layer chips at 100 to 107 degrees Celsius against a 110 degree automatic shutdown, because a fixed fan-speed override set a few days earlier for noise had switched off the fan algorithm entirely. A storage disk had been dead for a day and a half, with nothing to ask about it except the drive’s own self-assessment, which read PASSED. And a power event had rebooted a rack, leaving two hosts without remote management dark.

A drive can be physically gone for nine minutes and every health check stays green

Mon, 07 Sep 2026 00:00:00 +0000

The cluster is built the way most performance-minded Ceph clusters are built. Each storage daemon keeps its bulk object data on a spinning disk and its metadata database on a fast solid-state device. That split has a consequence nobody had connected to the earlier 38-hour blind disk. The routine health signals a storage daemon emits, its own up or down state, its heartbeat with its peers, the completion of a shallow scrub, are all satisfied by the metadata. Pull the platter out and the metadata is untouched. The daemon carries on reporting itself healthy, and a shallow scrub of the affected placement groups completes cleanly, because a shallow scrub validates metadata rather than object data.

Four monitor bugs in one week, and the silent ones were the dangerous half

Mon, 07 Sep 2026 00:00:00 +0000

Writing monitors quickly, under pressure, during live incident response produced four broken monitors in a single week. The four are worth listing because they fail in two different ways. The first parsed a column out of a cluster status command and reported zero inactive placement groups while five were blocked and client I/O had stopped. The second was written as a shell variable holding a whole remote-login command string, which works under bash and, under the environment’s actual default shell, becomes a single nonexistent command name, so it failed on every run and checked nothing. The third pinned the log it watched to the current date, so at midnight a perfectly healthy 24-hour job triggered an alarm. The fourth was a firmware-event watcher whose own login to the management controller wrote an event-log entry, which the next poll dutifully reported as a new event, forever.

A UPS running its own self-test looks exactly like a real power event, and nothing was watching UPS state at all

Mon, 07 Sep 2026 00:00:00 +0000

A UPS doing exactly what it is supposed to do, running a scheduled self-test every few weeks, raised the loudest alarm available to it: imminent shutdown of the storage host that everything else in the fleet depends on. Nothing was wrong. The monitoring plugin saw a battery-discharge state, concluded power had been lost, and started the shutdown countdown message.

False alarms are corrosive in one specific way, and this one is a textbook case. It teaches the reader to dismiss the next one, which may be real. The fix is not to stop watching for battery state. It is to stop treating battery state as the alarm. A UPS reports why it transferred to battery, and a self-test reason is a different value from a real utility loss. The rule now is that an on-battery state alone means nothing; it must be paired with a transfer reason that indicates a genuine event.

Building a dead-man's switch that only proves itself by being killed

Sun, 06 Sep 2026 00:00:00 +0000

The design rule is one sentence: an alarm that travels through the machine it watches is not an alarm. The day before the founding brief was written, the primary storage server had been dark for hours with nothing said. So the notification server was deliberately put on a different host, and the alerting engine was set to post a permanent heartbeat alert to it every minute. If that heartbeat stops arriving for about ten minutes, the notification host pages on its own, without needing anything from the fleet’s metrics stack.

A monitor that logs into the thing it watches becomes part of the fault it reports

Wed, 09 Sep 2026 00:00:00 +0000

Two incidents that week had found monitors that lied about the system. This one found a monitor that changed it. The UPS management host started reporting that it had lost communication with each of the four UPS units in turn, which is precisely the shape of a real, spreading electrical problem. There was no electrical problem. A probe was checking UPS state by opening an SSH session to the management host roughly every 15 seconds, and each of those logins is not a cheap operation: it creates a full user session, runs the login machinery, and tears it back down. That churn was enough to starve the UPS driver processes running on the same host until they timed out, one at a time.

A monitoring tool with zero monitors configured

Mon, 07 Sep 2026 00:00:00 +0000

After a week spent building infrastructure monitoring quickly and under pressure, the team stopped and audited what was actually covered. The answer was uncomfortable. Hosts, disks, switches and the storage cluster were well watched, and the services those machines exist to run were almost entirely unwatched. Only a handful of the running application containers had any check at all. Worse, an uptime-monitoring tool deployed some time earlier, which from the outside looked exactly like service coverage, turned out to have no monitors configured inside it. It was running, healthy, and checking nothing.

The hard drives were hiding their worst counters, and a reboot's letter shuffle paged seven false alarms

Thu, 10 Sep 2026 00:00:00 +0000

Two unrelated mistakes compounded into one class of failure on the same day. The first: on the fleet’s most common hard drive model, the SMART attribute table that everyone reads simply does not carry a field for read retries or uncorrectable reads. Those counters exist, in a separate ATA device-statistics log that most tooling never asks for, and one drive in the fleet turned out to be carrying hundreds of thousands of hidden read retries behind an attribute table that looked spotless. The data had always been there; nothing had ever queried it.

An alert fired continuously on the normal state of a design it misunderstood, and was removed rather than tuned

Fri, 11 Sep 2026 00:00:00 +0000

A monitoring rule can read its target correctly every single time and still misrepresent the system. This one did. The fleet advertises internal routes over a mesh VPN from two redundant routers, and the metric the alert watched reports which of them is currently the elected primary. In a healthy, correctly redundant pair, exactly one router is not the primary, at all times, forever. The alert had been written by someone whose mental model of “advertising the route” did not match how the failover software actually behaves when there are two of them. So it reported the design working as intended as a fault, continuously, from the day the second router was added.

An alert redesigned by hand three times before it survived independent review

Fri, 11 Sep 2026 00:00:00 +0000

The original alert said one thing and only one thing: the cluster has been in a warning state for 15 minutes. It said that whether one disk was slow or three were failing. A second problem arriving on top of the first produced no visible change at all, so the alert quietly turned into a standing reminder that no longer described what was wrong. Every person who read it had to go and find out for themselves what it meant, every time, for the life of the system.

A folder permission left from a partial deploy silenced the entire alerting path for 98 minutes

Fri, 11 Sep 2026 00:00:00 +0000

A deploy script wrote a shared secrets folder with a restrictive mode and intended to relax it thirty lines later in the same run. Anything that read the folder in the window between those two lines would fail. The alerting engine’s own container did, because it needed a credential from there to deliver anything at all. For 98 minutes the fleet had no alerting. Not degraded alerting, none. No heartbeat, no warning, no critical page.

A written-down cluster quorum threshold became wrong, in the dangerous direction, the day a fourth member joined

Sun, 13 Sep 2026 00:00:00 +0000

A number typed once into a monitoring rule, correct on the day it was written, became a landmine the moment the system it described changed shape. The rule paged when fewer than two members of the virtualisation cluster were answering, which is right for three members. A fourth member joined. Quorum for four is three, so a two-of-four cluster is genuinely frozen, and the rule would have said nothing at all. That is the dangerous direction. A rule that fails to fire when it should is more dangerous than one that fires too often, because nothing about it looks broken from the outside.

Shipping an alert before the thing it watches exists pages a human about your own unfinished work

Sun, 13 Sep 2026 00:00:00 +0000

The rule was not buggy. It did exactly what it was designed to do. It detected exactly the condition it was written to detect: nothing was publishing the health-check data it expected. That was true. The collector that would publish that data had not been built yet, because it was still waiting on review, and the rule had been committed alongside it as part of the same piece of work.

Creating an inventory record for a new host starts paging about it before it is built

Tue, 15 Sep 2026 00:00:00 +0000

In this fleet, the inventory system is not documentation. NetBox drives service discovery. Writing a device record with a monitoring class assigned is therefore the same action as arming that device’s alarm, and it takes effect within a minute or two. Creating the record for a new wall-mounted display was a purely administrative act in the mind of whoever typed it. Eleven minutes later, before the monitoring agent that would answer the scrape had even been installed, the critical host-down alert was paging about a build that was going perfectly well.

An abandoned wait loop locked the only Grafana account out for a full day, by re-arming its own lockout

Tue, 15 Sep 2026 00:00:00 +0000

The mechanism here is counterintuitive enough to be worth sitting with. Something tripped the dashboard account’s failed-login lockout once. After that, every request from the abandoned readiness loop was rejected specifically because of the lockout. Each rejection counted as another failed attempt, which re-armed the same lockout. Twelve times a minute, for a day. The loop’s credentials were entirely correct, proven the moment the lockout finally cleared. Resetting the password would have changed nothing at all, because the lockout was a property of the account rather than of the secret being presented to it.

Silencing a host by name during maintenance muted almost nothing the maintenance actually caused

Thu, 17 Sep 2026 00:00:00 +0000

Scoping a maintenance silence to the name of the host you are about to power down is the obvious thing to do. It is what the existing runbook said, and it had worked once before. On 20260917 it silenced almost nothing that the maintenance actually produced. Eight alerts paged.

The reason is structural rather than a mistake anyone made on the day. Alerts exist at more than one layer of aggregation, and the label that says who an alert is about reflects the layer the rule lives at. A rule watching the storage cluster names the cluster as its subject, because from the cluster’s point of view the affected entity is the cluster. A rule watching the network layer names the switch, because the switch is the thing reporting; the host that went down appears only as a neighbour reference buried in the alert’s detail. Neither of those is wrong. Neither of them matches a silence keyed to the host’s device name.

Forty-one of three hundred and forty-seven alerting rules carried no instruction for whether to act

Sun, 20 Sep 2026 00:00:00 +0000

Partway through building the fleet’s alerting rules, a convention was adopted. Every rule carries a label saying whether it is there to inform a diagnosis or to page a human. Everything written after that day followed it. Nothing went back to check whether everything written before it did too, because there was no mechanism that would force that check. A missing label is invisible: the rule still evaluates, still fires, still routes somewhere. It just carries no answer to the one question any tooling reading it needs to ask.

A monitoring runbook's own numbers were stale prose, not live truth

Sun, 20 Sep 2026 00:00:00 +0000

None of these mismatches were dangerous on their own. A runbook saying seven dashboards when there are fifteen costs nobody anything directly; it is simply a sentence written when the count was seven, which nobody had reason to revisit as integrations were added one at a time. The design document describing a dead-man’s-switch response time as about ten minutes was similarly innocent. Ten minutes was the target, and the measured value of just over thirteen lived in a different document written by a different pass.

A capacity alarm that compared totals paged every night, and a notifier that did not log its own failures went silently blind

Wed, 29 Jul 2026 00:00:00 +0000

Two mistakes in one script arrived at the same destination, an alarm system nobody could trust, from opposite directions.

The first was a capacity guard that compared the total size of the dataset it intended to copy against the free space on the destination. That comparison is correct on the first night and meaningless on every night after, because a mature backup rotation works precisely by having nearly everything already present at the destination, which is also why little space is free. So the guard fired every single night, for weeks, while the actual work each night was fifteen seconds of new data. Any alarm comparing one mostly-unchanging total against another mostly-unchanging total will settle into a permanent state and stop carrying information. The fix is almost always to alert on the marginal change rather than the total: what is new tonight, against what is free tonight.

Backrest's own error API caps at 21 entries, and an operator chased a fix that had already worked

Wed, 19 Aug 2026 00:00:00 +0000

Backrest’s operations API returns at most 21 error entries per run. This is a documented display limit and not a bug. But the summary reads as a count, and a count is what anyone looking at it will act on.

The effect on the work was almost backwards from the usual monitoring failure. Excluding a known-noisy file from a backup run removed one error, which freed a slot, which another genuinely failing file promptly filled. The reported total stayed pinned at exactly 21 across roughly 180 runs. Every exclusion added looked like it had accomplished nothing, when in fact each one had worked exactly as intended. This was not a false alarm; it was a false absence of improvement, which is harder to notice because nothing about it feels wrong. Checking the true count for one run directly gave 39, nearly double what the API reported.

You cannot see your own front door from inside your own home

Thu, 20 Aug 2026 00:00:00 +0000

A renumbering left an administrative login page answering on a public address for most of a day. Nothing in the fleet noticed, and the reason is structural rather than an oversight. Every monitor ran from inside the network. From inside, a probe against the site’s own public address returns the same result, no answer, whether that address is correctly firewalled to silence or is serving an unauthenticated administration panel to the entire internet. Those two states were genuinely indistinguishable from every vantage point the fleet possessed. It was found because someone happened to be on an unrelated external host that morning and looked.

Two measuring tools lied about a healthy network during a live firewall upgrade

Fri, 11 Sep 2026 00:00:00 +0000

Both quirks are worth knowing on their own. The first: the ping utility on macOS interprets its timeout flag in milliseconds, where the common convention on Linux is seconds. A value copied across from a Linux habit therefore tests with a timeout roughly a thousand times shorter than intended, and the tool duly reports losses that are not there. The second: on that firewall platform, process-listing tools run without elevated privileges cannot see any process owned by another user. A check for a background software update running as root therefore returns nothing, indistinguishable from the update having died.