Two measuring tools lied about a healthy network during a live firewall upgrade
- date
- 20260911
- what happened
- During a production firewall pair upgrade on 20260911, two measurement tools produced confident, wrong evidence of an outage that was not happening. A ping utility on a laptop reported multi-second connectivity gaps that were an artefact of its timeout flag being read in milliseconds rather than seconds. A process check run as an unprivileged user reported that a background update process was not running, because that platform hides other users' processes from unprivileged listings.
- what it cost
- Real time spent investigating a network problem that did not exist, during a live, sensitive maintenance window on production firewall hardware.
- what changed
- The standing rule that when a measurement disagrees with an independent existing signal, the measuring tool is suspected first, by checking whether its own output is internally consistent.
- the check now
- Run network probes from a host whose platform behaviour is confirmed rather than assumed, or corroborate a suspected outage against an independent signal before acting. Use an elevated command, or a signal that does not depend on process visibility, to check for a process running as another user.
Both quirks are worth knowing on their own. The first: the ping utility on macOS interprets its timeout flag in milliseconds, where the common convention on Linux is seconds. A value copied across from a Linux habit therefore tests with a timeout roughly a thousand times shorter than intended, and the tool duly reports losses that are not there. The second: on that firewall platform, process-listing tools run without elevated privileges cannot see any process owned by another user. A check for a background software update running as root therefore returns nothing, indistinguishable from the update having died.
Either quirk on a quiet afternoon is an annoyance. Both of them arrived during a live upgrade of a production firewall pair, where a false signal of trouble could easily have prompted rolling back a change that was in fact working correctly. That is the cost. The wrong number arrived at the moment it was most likely to be believed and acted on, which did more damage than the number itself.
The habit that came out of it is a specific form of scepticism, and it is cheap. When a measurement produces a surprising result, check the measurement against itself before checking the system. Does the log’s recorded line count match the elapsed time it claims to cover? Does the command account for the privilege level it is running at? In this case the line count was the tell: a ping log covering ten minutes had far too few lines in it to have been sampling at the rate anyone thought it was.
A confident wrong reading usually comes from an assumption inside the observing tool, not from the system being observed.
Source: node0 lessons v0.1, lesson 2.22. Sanitized: checklist v0.1, 20260921; names pass only; voice pass 20260921. Part of oznog.com/node0.