A failed fetch still exits zero, and already converged can mean still stale
- date
- 20260910
- what happened
- On the fleet build host, a Nix build with a refreshed flake silently falls back to the locally cached tree when the git fetch fails, printing a warning and exiting zero. Every layer above read that as success: the hourly pull-deploy convergence logged already converged, and a manual rebuild printed Done while building the old store path. It happened twice in one hour, once from a bond going down after a NIC swap and once from a wrong key path.
- what it cost
- Up to an hour of believing a host had converged to the latest configuration when it had not, on a fleet whose safety model depends on that convergence being real.
- what changed
- Pull-deploy now reads the branch head directly with a remote ref listing and requires the flake metadata to resolve to that exact revision, failing explicitly with not converging rather than quietly building from cache. A manual switch is verified by comparing the printed store path against the build host's result.
- the check now
- Compare the resolved git revision or the store path, never the exit code or the log line.
Any pull-based or GitOps-style deployment built on Nix should read this closely. A refresh is a request, not a guarantee. A tool that cannot reach its source of truth has to choose what to do instead, and Nix’s choice here, proceed with the cached tree and warn quietly, is reasonable in isolation. It stops being reasonable the moment other automation trusts the exit code alone, which is what every convergence loop built on top of it does by default.
The two triggers are worth noting because neither looks like a build problem. One was a network bond that came down after a NIC swap, so name resolution failed. The other was a key path that no longer existed, so the git server refused the connection. In both cases the build succeeded, the words on screen were success words, and the machine kept running the configuration it already had.
The fix generalises past Nix to almost any convergence-based deployment tool. What proves a change landed is never the tool’s own verdict on itself. It is an independent fact the tool cannot fabricate by falling back quietly: a resolved revision hash, a store path, a build identifier. Make the deployment compare that fact against what was asked for, and make the mismatch an explicit failure with its own words. That way the log line saying nothing to do means the same thing every time it appears.
Source: node0 lessons v0.1, lesson 6.10. Sanitized: checklist v0.1, 20260921; hostname, git server domain, key paths; voice pass 20260921. Part of oznog.com/node0.