Remote access on a storage box, and the redesign that survived a lab-caught wrong assumption
- date
- 20260905 (the outage that triggered the redesign) to 20260911 (production cutover)
- what happened
- Node0's remote-access subnet routing, which lets Christoph reach internal networks from anywhere without opening a general hole in the firewall, had accumulated on unraid1, a storage server that happened to run the mesh client, rather than on a host built to be a router. A routine maintenance window on that storage box on 20260905 took internal DNS down fleet-wide, because the only resolver reachable over the mesh was hosted on the box that was down.
- what it cost
- A fleet-wide DNS outage for the duration of the maintenance window, and the discovery that remote access depended entirely on a host that had not been chosen, or maintained, with that responsibility in mind.
- what changed
- Subnet routing moved onto both firewalls independently, each running its own copy with no shared state between them. An early version of the design assumed a firewall failover would be survivable because the shared address would move and packet-level state sync would carry the connections. A lab test built specifically to check it proved that false, so the shipped design routes per node, with its own address translation, and never relies on the pair's failover for this traffic.
- the check now
- Prove any assumption about stateful failover under a tunnelling protocol in a lab, with a forced asymmetric failover, before depending on it. Do not host a fleet-critical routing or resolution service on a box whose primary job is something else.
A convenience that grew organically, running remote-access routing on whichever box happened to be online and capable, turned into a single point of failure for the whole site’s DNS on 20260905, the day that box needed routine maintenance. Nothing had decided that the storage server should hold the remote-access path. It simply already ran the client, and each new advertised route was individually reasonable.
The redesign that followed did the disciplined thing. It questioned its own core assumption in a lab before trusting it in production, and the assumption turned out to be wrong in an interesting way. The intuitive design was to let the firewalls’ existing failover mechanism carry the remote-access routing too, with connection state synced between the pair so that a failover would be invisible. The plan had asserted, in writing, that this case was “asymmetric but survivable”.
A targeted lab test proved it was not. The tunnel protocol underneath does its own cryptographic peer verification and does not care what the firewalls think the current state is. A reply arriving from the wrong firewall, even one presenting the correct shared address, is silently discarded by the client, because the client only accepts a reply from the peer address it expects. There is no error, no retry that helps, just a session that stops.
The corrected design routes per node, each firewall independently, with its own address translation, deliberately never depending on the pair’s shared address or its state sync for this traffic. The broader lesson: when a redundancy scheme sits underneath a protocol that does its own connection or peer verification, check whether that protocol will accept a failed-over connection at all. State synchronisation at the network layer does not make a cryptographic peer check agree with you.
Source: node0 lessons v0.1, lesson 4.13. Sanitized: checklist v0.1, 20260921; mesh addresses, advertised routes; voice pass 20260921. Part of oznog.com/node0.