A restrictive permission mode that looks like a broken account, not a broken directory
- date
- 20260830, cause found 20260904
- what happened
- On a freshly built NixOS host, SSH key-based login worked for the root account and was refused for every non-root account using the identical key, which reads like a per-account authorisation problem. The cause, found days later on a repeat during a second host rebuild, was that the SSH configuration directory and the wider system configuration directory had been restored from an archive at a mode only their owner could enter, so no non-root session could read its own authorised keys file.
- what it cost
- Misdirected troubleshooting on the first occurrence: account configuration, key fingerprints and server settings were all checked and re-checked, and all were correct. The second occurrence was diagnosed quickly once the pattern was known.
- what changed
- The root cause was traced to the archive used to restore a host's SSH identity during a rebuild. The archiving tool's default mask recorded an overly restrictive mode on the directory entries, not only on the key files inside them, and extracting as root silently overwrote modes an earlier setup step had set correctly. Re-setting the directory modes after extraction, before rebooting, is now a required step. Rebuilding the archive so it cannot carry the restrictive mode at all is still open.
- the check now
- When key authentication works for root and fails identically for every other account, compare the mode of the SSH configuration directory and its parent against a known-good host before re-checking accounts or keys.
The single most misleading fact in this incident is that root’s own SSH login worked perfectly, with the exact same key that failed for every other account. That points every reasonable instinct at the failing accounts rather than at something both accounts share, which here was the directory tree itself.
Root’s process bypasses most filesystem permission checks, so a directory that only its owner can enter is invisible to root and impassable to everyone else. A non-root SSH session has to traverse the system configuration directory and then the SSH directory to read the file listing its own authorised keys. If either directory refuses traversal, the session is rejected before any key is ever compared. The server logs nothing that looks like a permissions problem, because from its point of view the user simply had no acceptable key.
Root’s success did not merely fail to reveal the cause. It actively hid it, by providing a working control case that appeared to exonerate the whole shared path.
The fault traced back two layers, to a backup and restore step. Its archiving tool recorded a restrictive mode not on the sensitive files, which is expected and correct, but on the directories containing them, which is not. Extracting that archive as root undid what an earlier, correct step had set.
The generalisable diagnostic follows a simple test. Whenever one privileged account behaves correctly while every unprivileged account fails identically at the same operation, suspect something the two touch differently by privilege level. Filesystem permissions are the most common such thing, and they are cheap to compare against a host that works.
Source: node0 lessons v0.1, lesson 4.19. Sanitized: checklist v0.1, 20260921; host names; voice pass 20260921. Part of oznog.com/node0.