Network and edge on Oznog

CARP split-brain that was two bad cables, not bad config

Mon, 17 Aug 2026 00:00:00 +0000

A pair of firewalls that are supposed to fail over into each other instead both decided they were in charge at once. That is the one CARP failure (Common Address Redundancy Protocol, the mechanism that lets two firewalls share one gateway address). It looks completely normal from a status page: every shared address answers, every check passes, and there is quietly no redundancy at all. It happened on 20260817, during bench bring-up, on two of the three shared addresses.

The virtual-MAC fault on the circuit: a two-day diagnosis with the provider

Tue, 18 Aug 2026 00:00:00 +0000

A firewall failover is supposed to be invisible to everyone outside the building. A virtual MAC address, shared by both firewalls, exists precisely so that an internet provider never has to notice which physical box is answering. Node0’s provider noticed anyway. Its edge equipment bound each public address to whichever MAC address last spoke from it, and it never relearned a virtual one automatically. Every address carried by the pair went dark after a failover and stayed dark.

The admin login page open to the internet, closed ten minutes after it was seen

Thu, 20 Aug 2026 00:00:00 +0000

A firewall’s own login page ended up reachable from anywhere on the internet on 20260820, for a time nobody recorded, and was closed about ten minutes after it was noticed, as the fallout of an otherwise routine address renumber. A rule written for one address kept doing something after the service it was written for moved off that address, just not the thing it was written to do. Everything else in the renumber, the outbound translation, the proxy bindings, the second firewall’s own filter rules, had been updated together. One set of inbound rules on the first firewall had not.

Kea DHCP high availability: five defects before it actually failed over

Mon, 17 Aug 2026 00:00:00 +0000

Setting up redundant DHCP sounds like flipping one switch. Enable high availability on both firewalls, done. It took five separate, independently silent defects before Kea’s hot-standby model actually did what its name promised, and every one of them looked fine from the ordinary places anyone would check. The interface showed enabled. A process was running. A status page reported no error.

Each defect was a default that quietly did the wrong thing in a failover context specifically. The firewall’s own lightweight DHCP daemon still held the port Kea needed, and the log message never named it as the cause. The control agent, which the firewall needs to write the failover hook into Kea’s configuration at all, was pointed at the same port as the hook itself. A spurious warning tempted a change of socket mode that would have made the server unable to hear broadcast discovery packets entirely. An autocollect convenience replaced the pair’s shared gateway address with one firewall’s own, so a failover would have handed clients a route to a dead firewall. And ten days later, in production, the config-sync process copied the setting by which each node names itself, so the backup could not find itself in its own peer list.

The management switch that arrived from a previous life, locked

Tue, 18 Aug 2026 00:00:00 +0000

Buying used enterprise network gear carries the same risk as buying used enterprise servers. Configuration left by a previous owner can still be active and enforcing itself. This switch arrived on 20260818 completely unmanageable through any of the normal channels, including the vendor’s own purpose-built recovery tool, because an access control list from a previous deployment was still enforcing itself against source addresses that had not existed for years.

The diagnostic tell worth keeping is the distinction between refused and filtered. A service that is off or has crashed answers a connection attempt with a reset, immediately and unambiguously. A service that exists but is blocked by policy answers nothing at all, while everything else on the device, in this case plain ping, keeps working normally. Four hosts on different segments all saw the same thing, which ruled out anything local to one of them.

A default OpenWrt access point is a rogue DHCP and DNS server

Tue, 18 Aug 2026 00:00:00 +0000

The access point’s most famous default-configuration trap, a DHCP server that would fight the real one, was anticipated correctly. The whole deployment on 20260818 was staged around avoiding it: bring the device up isolated on a throwaway host, strip its dangerous defaults there, and only then connect it to the live network. That discipline worked exactly as intended.

What slipped through was a second service riding on the same daemon as the one everyone was watching. On OpenWrt, one process serves both DHCP and DNS. Turning off DHCP in the interface turns off the leases. The process stays up and keeps answering DNS queries on the access point’s own address, silently, to anyone who asks. Nothing was configured to ask, so nothing broke, and it went unnoticed for days.

The Intel chip that eats its own DHCP replies

Fri, 28 Aug 2026 00:00:00 +0000

Some hardware faults look exactly like software faults, and this is one of the cleaner examples. An Intel network chip’s manageability filter, present because the chip supports remote management features Node0 never provisioned, ate every DHCP server reply leaving that port. Nothing was visible to the operating system, and nothing was visible to a packet capture run on the same box, because the drop happened in hardware below where either could see it. The sending host shows the frame handed to the driver and then it simply does not exist.

A VLAN renumber that never touched a live address

Thu, 27 Aug 2026 00:00:00 +0000

This is the lesson that is boring because it worked. Nothing broke on 20260827, no outage happened, and none of this is a postmortem. It is here because the discipline it represents is what kept the much larger migration a day later from breaking things it easily could have broken.

Renumbering plans naturally want to update everything at once, including DNS and live addressing, because the new numbers are the state the plan describes. Holding two versions of the truth in your head is uncomfortable. The temptation is strongest for the small stuff: a record that is obviously going to change anyway, a host that is obviously moving next week. Each of those edits is individually harmless and collectively produces a network where nobody can tell which addresses are real.

Cutting a flat 135-host network to eighteen VLANs in one day, and the DNS outage that came out of it

Fri, 28 Aug 2026 00:00:00 +0000

Migrating an entire site’s network in a single day, rather than gradually, is the kind of decision that either goes very well or produces one very specific, very confusing failure. On 20260828 Node0 got a bit of both. The migration itself succeeded. Every online host moved, in an order chosen so that the services everything else depends on, DNS above all, stayed reachable the whole time.

The one failure it produced was subtle precisely because everything anyone knew to check reported fine. The affected server was up, its service was running, and a direct query against it succeeded. What was broken lived entirely in the switch’s own memory of where that server’s address used to live, pointing at a MAC address the server no longer used. A switch relearns an address when the host actively sources traffic from it, and a secondary nameserver mostly only ever replies. It had nothing to say unprompted, so the switch kept sending off-subnet packets to an address that had moved.

Why cross-rack switch pairing sounded right and was wrong

Mon, 17 Aug 2026 00:00:00 +0000

This is a design correction rather than an incident, and it is here because it changed a real decision on 20260817, before that decision became expensive to reverse: a long high-speed cable run between racks that the existing plant could not support anyway.

The earlier reasoning had genuine appeal. Spread a switch pair across two racks, using multi-chassis link aggregation so a host’s two uplinks land on two switches that behave as one. Then a host keeps a working link even if its own rack’s network hardware fails entirely. Written down that way it sounds like more redundancy for the same money.

One switch generation, one silent 2.5 gigabit ceiling

Mon, 14 Sep 2026 00:00:00 +0000

A whole class of network adapters in the fleet ran at a fraction of their rated speed, and nothing about it was visible from either end. The adapter reports its full capability, the switch reports its own port type, and only the actual negotiated rate, buried in the interface’s link state, tells the truth. The root cause is refreshingly simple once found. An older switch generation simply does not implement the in-between speed, so both sides fall back to the highest rate they share, which is 1 gigabit.

Zone transfers need a shared key, not just a source-address rule

Tue, 18 Aug 2026 00:00:00 +0000

DNS secondaries are supposed to be a solved, boring problem. Point a secondary at a primary, authorise it, and the zone data replicates. The authorisation mechanism that looks obvious is an allow list: permit only the secondary’s own known address to request a transfer, and refuse everyone else.

On 20260818 that turned out not to be reliable on this particular DNS server software, for a reason that has nothing to do with the network. A host with more than one address does not necessarily use the address you expect when it opens an outbound connection, and this software was not consistent about which of its own addresses it sourced a transfer request from. The primary refused the ones it did not recognise. Both addresses were the secondary’s, both were legitimate, and no packet was lost or misrouted. The rule simply could not express the thing it was trying to express.

Remote access on a storage box, and the redesign that survived a lab-caught wrong assumption

Sat, 05 Sep 2026 00:00:00 +0000

A convenience that grew organically, running remote-access routing on whichever box happened to be online and capable, turned into a single point of failure for the whole site’s DNS on 20260905, the day that box needed routine maintenance. Nothing had decided that the storage server should hold the remote-access path. It simply already ran the client, and each new advertised route was individually reasonable.

The redesign that followed did the disciplined thing. It questioned its own core assumption in a lab before trusting it in production, and the assumption turned out to be wrong in an interesting way. The intuitive design was to let the firewalls’ existing failover mechanism carry the remote-access routing too, with connection state synced between the pair so that a failover would be invisible. The plan had asserted, in writing, that this case was “asymmetric but survivable”.

A NAT rule with a misspelled field name took public services down for six minutes

Sun, 06 Sep 2026 00:00:00 +0000

An API that accepts a request and reports success is not the same thing as an API that understood the request, and this is a clean demonstration of the gap. Two field names, chosen by reasonable guesswork rather than checked against the API’s actual schema, were close enough to look right and different enough to be dropped without comment. What got stored was a rule missing the two fields that mattered most, the destination and the port.

A resolver that lied by omission: healthy on every check while replication silently stopped

Sun, 20 Sep 2026 00:00:00 +0000

Three different signals said this DNS cluster was healthy on 20260920, and all three were describing the wrong thing. A connection status flag was true, because the connection between primary and secondary genuinely was open. A replication counter matched across every node, because it tracked the zone data, the actual DNS records, which travel by a completely different and entirely unaffected mechanism. And the dashboard reduced several fields into one word, healthy, which hid the one field that mattered.

A DNS helper's own safety feature nearly destroyed the wrong record twice

Tue, 15 Sep 2026 00:00:00 +0000

Some safety features fail by being too helpful, and this is a clean pair of examples on the same mechanism, four days apart. A DNS server offered a one-step convenience. It writes the forward record and the matching reverse record together, and later deletes them together too. Most of the time that is exactly what you want, which is why it is the default.

The convenience assumes an address has exactly one name that owns it. That is false for any shared ingress point, where several service names deliberately resolve to the same address, and it is false for any address that gets reused after a device retires. Both of Node0’s incidents came from exactly that false assumption, from opposite directions. On 20260915 the convenience wrote. Adding a new service name to a shared address took the reverse record away from the ingress that actually owns it, because the mechanism treats the reverse record as belonging to whichever forward record was written most recently. On 20260919 it nearly deleted. Removing a retired device’s record, minutes after its old address had been handed to a new device, would have taken the new device’s reverse record with it.

A VPN client on a laptop that quietly captured Node0's own routes

Sun, 30 Aug 2026 00:00:00 +0000

A consumer VPN client, running for an entirely unrelated reason, general internet privacy on a personal laptop, reached into that laptop’s routing table and sent every request bound for the internal network through itself, with no warning and no obvious toggle responsible.

The resulting symptom on 20260830, every internal destination down while the local network and the internet worked fine, is exactly the shape a real firewall failure would produce. The coincidence that a genuinely unrelated firewall problem had happened earlier the same day made the wrong diagnosis more tempting rather than less. The evidence appeared to corroborate a theory that already had a plausible cause attached to it.

An SSH security default that pages as a network outage

Fri, 21 Aug 2026 00:00:00 +0000

The trap here is how convincingly the symptom mimics an unrelated class of problem. A security feature, on by default in current OpenSSH and therefore on most current Linux distributions, blocks a source address for a period of time after a small number of failed login attempts. The client sees “connection timed out”, which gives no hint at all that this is what happened. It reads exactly like a firewall rule, a routing problem, or a host that is simply not there.

A restrictive permission mode that looks like a broken account, not a broken directory

Sun, 30 Aug 2026 00:00:00 +0000

The single most misleading fact in this incident is that root’s own SSH login worked perfectly, with the exact same key that failed for every other account. That points every reasonable instinct at the failing accounts rather than at something both accounts share, which here was the directory tree itself.

Root’s process bypasses most filesystem permission checks, so a directory that only its owner can enter is invisible to root and impassable to everyone else. A non-root SSH session has to traverse the system configuration directory and then the SSH directory to read the file listing its own authorised keys. If either directory refuses traversal, the session is rejected before any key is ever compared. The server logs nothing that looks like a permissions problem, because from its point of view the user simply had no acceptable key.

Two files, one name lookup: reconciling six laptops' worth of SSH trust

Sun, 30 Aug 2026 00:00:00 +0000

Trust on first use, SSH’s default model for remembering which cryptographic key belongs to which host, works fine for one machine talking to a stable set of hosts. It works badly for a small fleet of administrative laptops that have each independently accumulated a partial, diverging record of the same infrastructure: between 95 and 159 entries each, no two alike, and no way to tell a deliberate entry from an accident.