A DNS helper's own safety feature nearly destroyed the wrong record twice
- date
- 20260915 and 20260919
- what happened
- The DNS server's API offers a convenience that writes or deletes an address's reverse record alongside its forward record. On 20260915, adding a new service name that shared an address with an existing shared ingress point used that default, which replaced the ingress's own reverse record with the new name. On 20260919, deleting a retired device's record minutes after its former address had been reassigned would have deleted the new occupant's reverse record along with the old occupant's forward one.
- what it cost
- On 20260915, a shared ingress address answered reverse lookups with the wrong service name until it was found and restored. On 20260919, nothing, because the guard added after the first incident caught it.
- what changed
- The in-house DNS helper script now takes an explicit flag for a name that must not own an address's reverse record, and on deletion it reads the current reverse record first and skips it when it names a different host.
- the check now
- Never add or delete a record on a shared address using a write-both convenience without first reading what the reverse record currently says. Treat replace semantics as a footgun on any address more than one name might resolve to.
Some safety features fail by being too helpful, and this is a clean pair of examples on the same mechanism, four days apart. A DNS server offered a one-step convenience. It writes the forward record and the matching reverse record together, and later deletes them together too. Most of the time that is exactly what you want, which is why it is the default.
The convenience assumes an address has exactly one name that owns it. That is false for any shared ingress point, where several service names deliberately resolve to the same address, and it is false for any address that gets reused after a device retires. Both of Node0’s incidents came from exactly that false assumption, from opposite directions. On 20260915 the convenience wrote. Adding a new service name to a shared address took the reverse record away from the ingress that actually owns it, because the mechanism treats the reverse record as belonging to whichever forward record was written most recently. On 20260919 it nearly deleted. Removing a retired device’s record, minutes after its old address had been handed to a new device, would have taken the new device’s reverse record with it.
The second one cost nothing only because the first had already prompted building a check into the wrapper tool: read the reverse record, see whether it still names the host being removed, and leave it alone if it does not.
The transferable habit for anyone scripting against a similar API: a convenience that writes two records because they usually go together is only safe if you also handle the case where they no longer do. For a shared or reassigned address, that case is routine, not an edge.
Source: node0 lessons v0.1, lesson 4.16. Sanitized: checklist v0.1, 20260921; service and host names, tool path; voice pass 20260921. Part of oznog.com/node0.