A resolver that lied by omission: healthy on every check while replication silently stopped
- date
- 20260920
- what happened
- An unattended container update recreated the primary DNS server's container overnight, applying a non-root user setting that had sat dormant in the template for months. The new user could not write the log files the previous root-run container had created, and the service exited. The manual restart brought the primary up on a newer release than its two secondaries.
- what it cost
- About thirty minutes with the primary down, with no fleet-wide impact because both secondaries kept answering every query, by design. Then an unknown period followed during which configuration, blocklists and policy stopped replicating entirely, while every liveness signal continued to report the cluster as healthy.
- what changed
- One secondary was upgraded to match, restoring replication immediately, proven by a stale sync timestamp updating the moment the versions matched. The primary's container was pinned to an exact version tag rather than a floating one. The standing decision (Christoph, 20260920) is version parity coordinated inside one maintenance window, members restarted one at a time, with the third left on the older release until the operating system's own package repository ships the match.
- the check now
- On any clustered service, monitor a signal that proves data actually landed, a last-synced timestamp, rather than a signal that proves a connection exists. Pin the exact version of a clustered service's image so no unattended process can move it.
Three different signals said this DNS cluster was healthy on 20260920, and all three were describing the wrong thing. A connection status flag was true, because the connection between primary and secondary genuinely was open. A replication counter matched across every node, because it tracked the zone data, the actual DNS records, which travel by a completely different and entirely unaffected mechanism. And the dashboard reduced several fields into one word, healthy, which hid the one field that mattered.
The real fault was a version incompatibility. A newer primary writes a configuration bundle that older secondaries cannot parse, so the thing that keeps every node’s behaviour identical, the blocklists, the policies, the settings, simply stopped moving. It showed up only in the secondaries’ own raw error logs, thrown and ignored on a timer, hundreds of times, while every higher-level view insisted nothing was wrong.
The chain that got there is worth noticing too. An unattended update mechanism was allowed to move the version of a clustered service; a setting nobody had used since it was added took effect the moment the container was recreated; and the restart that fixed the immediate outage is what introduced the silent one.
What is worth carrying elsewhere is the shape of the trap. A clustered system can have working liveness, meaning I can talk to my peer, and working partial replication, meaning this one specific thing keeps syncing, at the same moment its actual purpose has completely stopped. No dashboard built around “connected” and “in sync” as its top-line fields will show you that unless you know to distrust them and go read a timestamp instead.
Source: node0 lessons v0.1, lesson 4.15. Sanitized: checklist v0.1, 20260921; host names, release numbers; voice pass 20260921. Part of oznog.com/node0.