Two NICs on one installer subnet look exactly like a crashing host
- date
- 20260830 to 20260831
- what happened
- A Mac mini's built-in and USB Ethernet adapters both landed on the installer subnet during a NixOS bring-up. The same address flapped between two hardware addresses, and the install looked like a host crashing under load while behaving normally at idle. Separately, another Mac mini's SD card failed with ADMA write errors.
- what it cost
- A misdiagnosed install session spent chasing a crashing machine that was a routing artefact.
- what changed
- The rule became to single-NIC the installer first, before the first hang rather than after it. Installs now boot with the image copied to RAM, so a flaky USB stick's read errors stop mattering, and a card throwing ADMA errors is replaced rather than diagnosed further.
- the check now
- Bring every network interface but one down before starting an install on a multi-NIC host, and soak-test any new SD or USB media with a write and readback before trusting it in service.
Sustained load turned a routing quirk into a symptom that read exactly like failing hardware. During a NixOS install on 20260830, the machine was fine at idle and fell over the moment real data moved, which is the shape that sends everyone reaching for a hardware explanation.
The cause was that the host had two Ethernet interfaces, its built-in port and a USB adapter, and both took an address on the same installer subnet. Replies left by whichever interface the switch preferred at that instant. One address kept moving between two hardware addresses, and the switch’s forwarding table kept being rewritten. At idle there was no traffic to fight over, so nothing showed. Under a sustained transfer, pushing a NixOS closure (the complete set of packages a configuration needs) to the target, the session died repeatedly and looked like a crash.
The rule that came out of it is boring and cheap: bring every interface but one down before starting an install on a multi-NIC host, before the first hang, not after it. Installs also boot with the image copied to RAM now, so a marginal USB stick cannot produce read errors halfway through.
The same bring-up produced a useful contrast. A second Mac mini’s SD card, used for a small scratch mount, threw ADMA write errors, and a replacement card tested identically wrote clean. Some failures really are the hardware, and the fast way to tell which is a proven substitution rather than more diagnosis on the suspect part.
Source: node0 lessons v0.1, lesson 1.6. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.