Cutting a flat 135-host network to eighteen VLANs in one day, and the DNS outage that came out of it
- date
- 20260828
- what happened
- All 135 online hosts were moved in a single day from one flat network to their assigned VLANs, in a fixed order: a disposable test host first, then the storage box, the management plane, time and agent hosts, the dev boxes, the access point, the shared sandbox, with the DNS secondaries deliberately last so a resolver stayed reachable throughout. One host, carrying a secondary nameserver, moved from a single network card onto a bonded interface, which changes the MAC address the switch sees.
- what it cost
- About two and a half hours of DNS resolution failure for anything reaching that secondary from off its own subnet, while the host itself, and every check run directly against it, reported completely healthy.
- what changed
- The immediate fix was clearing the stale entry on both switch peers. The durable change was a documented diagnostic chain for this failure class, and a rule that any host moving onto a new physical interface must have every secondary address it carries checked for the same staleness, not just its primary.
- the check now
- After moving any host onto a new interface, query the switch for addresses it has not learned. A not-learned entry for a live host's address is a real outage needing a forced clear. The same entry for a retired address should be left to age out.
Migrating an entire site’s network in a single day, rather than gradually, is the kind of decision that either goes very well or produces one very specific, very confusing failure. On 20260828 Node0 got a bit of both. The migration itself succeeded. Every online host moved, in an order chosen so that the services everything else depends on, DNS above all, stayed reachable the whole time.
The one failure it produced was subtle precisely because everything anyone knew to check reported fine. The affected server was up, its service was running, and a direct query against it succeeded. What was broken lived entirely in the switch’s own memory of where that server’s address used to live, pointing at a MAC address the server no longer used. A switch relearns an address when the host actively sources traffic from it, and a secondary nameserver mostly only ever replies. It had nothing to say unprompted, so the switch kept sending off-subnet packets to an address that had moved.
No amount of checking the server itself would ever have revealed that. The fix, once diagnosed, is a single command to clear the stale entry, or a gratuitous ARP from the host. The diagnosis took two and a half hours because nobody thinks to distrust a switch’s address table while everything above it looks healthy.
The transferable habit: any address that only ever replies and never initiates traffic, a secondary nameserver, a passive health-check target, most load balancer virtual addresses, is at risk of exactly this staleness after any change to the interface it rides on. It needs an explicit check, because “the host is up” does not mean “the network path to it is”.
Source: node0 lessons v0.1, lesson 4.9. Sanitized: checklist v0.1, 20260921; host names, addresses, VLAN numbers; voice pass 20260921. Part of oznog.com/node0.