Rules we now run by

Most of the lessons on this site ended in a fix to one machine. These twenty-seven ended in a rule for whoever operates the fleet next, human or agent, because the mistake was not about a device at all. The first twenty-one were collected on 20260920 out of the six lesson themes; rules 22 to 27 were added on 20260922 from Christoph’s read of these pages, each from the lesson it cites. All were checked against the runbook that carries the fleet’s operating traps to confirm none of them was already written there, and added to it as a dated section. Each is stated as the instruction an operator is expected to follow, with the lesson it came from.

1. After any hard hang, cold-boot the host; do not try a warm reset, a bus rescan, or repeated reboots first. On this fleet a warm reset has re-triggered the identical fault every time it has been tried, across two unrelated failure classes, and only removing power has ever cleared either. Trying the warm path first costs real time for a result you can already predict. (Lessons 1.4 and 1.11.)
2. Identify a drive, board or card by its own serial number or CPU identifier, never by its current device letter, bay number or slot. Letters and bay numbers move, in one incident twice during a single fault, so any alert or diagnosis keyed to them misfires silently. Resolve to a serial before trusting a comparison across time or across a maintenance event. (Lessons 1.9 and 1.14.)
3. Before pulling, blacklisting or otherwise disabling a storage controller driver, trace exactly which host bus and which drives sit behind it on this specific machine. The mapping is not consistent across otherwise identical hosts here, and acting on the assumption came within one caught mistake of making a hypervisor’s root pool unbootable. (Lesson 1.13.)
4. A drive or SSD that passes a health check at rest is not cleared for service. Reproduce the condition the suspected fault actually needs, a sustained write load, a full scrub, a repeatable burn-in, before returning a part to service or condemning it for a claim. An at-rest pass has already returned two drives to production that failed again immediately. (Lesson 1.17.)
5. Bound every wait loop, or make it die with its own session. Any poll-until-ready loop started over a remote session must carry an explicit timeout or run somewhere it is guaranteed to die when the session ends. A correctly credentialled poller retrying against a target that is rejecting it for a rate or lockout limit can re-arm the very block it is tripping, turning a transient failure into a permanent one. (Lesson 2.15.)
6. Build and confirm the producer before the rule, and treat creating an inventory record as an operational action. Confirm the metric or event series an alert depends on is live and populating before the rule is deployed, not merely that the code meant to produce it has been written, because a rule deployed against a producer that is not live yet pages about your own unfinished work, and a deployed rule does not undeploy itself once the gap closes. Where creating a record automatically starts monitoring the thing it describes, precede it with a scoped, time-limited silence or get the subject live immediately. Never leave a gap between the two, since that gap is exactly where the false page arrives. (Lessons 2.13 and 2.14.)
7. A managed configuration file is not a place to hand-edit, even under time pressure. Where a generator or validator owns a file, treat a direct edit to it as an incident to recover from rather than a shortcut; one hand edit took out two independent nightly backup destinations at once because both depended on a shared validated step underneath. Before editing any file whose header names it as generated, find the generator. (Lesson 3.17.)
8. Deletion is never in the same command as the transfer or the check it depends on. Transfer, verify by an explicit observed check such as a hash or a count, then delete, as three separate steps, with no exemption for small, quick or interactive work. One command that skipped this because it was small permanently destroyed data that every scripted equivalent in the same project had protected. (Lesson 3.1.)
9. A bulk operation’s own exit code or success message is a claim, not a proof, for anything destructive. Re-list or re-check the target state after any bulk delete or move and compare it against what was intended. This alone has caught a silent comment-character parsing trap, a permission-scoped API behaviour change, and a capacity guard that logged its own failure and then proceeded anyway. (Lessons 3.6, 3.7 and 3.9.)
10. A new host or service is not finished when it runs; it is finished when every fleet-wide check that should be watching it can actually see it. Run the verification, monitoring registration and credential sweep against the new component explicitly rather than assuming inclusion. One host’s backups, scrape target and credential rotation were each independently missed within two days of the same build. (Lesson 3.19.)
11. An SSH connection timed out is not evidence of a network problem. Current OpenSSH blocks a source address with an escalating delay after a few failed authentications from it, and the resulting error is indistinguishable from a routing, firewall or host-down fault. Check the per-source penalty setting on the target, or simply retry from a different source, before building an infrastructure theory. (Lesson 4.18.)
12. Before troubleshooting an apparent site-wide outage from your own machine, rule out a VPN client on that machine. A full-tunnel VPN can capture routes to the internal network while leaving the local gateway and the internet reachable, which looks exactly like a real infrastructure failure. Check which interface the route to an internal address actually uses, and whether an independent path to the same host still works. (Lesson 4.17.)
13. Root logging in successfully is not evidence that a permission problem is not the cause. Root bypasses most filesystem permission checks, so an over-restrictive mode on a shared directory blocks every unprivileged account while root sees nothing wrong. When one privileged account works and every other fails identically, compare permissions against a known-good host before re-checking keys and accounts. (Lesson 4.19.)
14. Never remove a firewall connection-tracking exception because the service it was named after is gone, without checking what else relies on the same mechanism. A rule named for one purpose can be load-bearing for an unrelated traffic pattern that happens to share the code path; here a remote-access exception was also what let bridge-network containers reach ipvlan-network containers on the same host. Search the host’s own incident history for the rule’s name before deleting it. (From the operating rules; the incident behind it, on the storage hub’s container networks, is not one of the published lessons.)
15. Never loop over a single-session management interface, and never kill a session mid-handshake. Several classes of power-hardware management card accept exactly one interactive session and fail silently under contention: a second session gets an empty response indistinguishable from a dead device, and fast repeated sessions, especially ones killed by a timeout rather than logged out, can wedge the interface for half an hour while the device itself works perfectly. Serialise access, let every session close cleanly, and treat one empty response as inconclusive rather than as proof. (Lesson 5.4.)
16. A secret rotation on a NixOS host is not proven by the affected units reporting active. Where a module renders configuration from a template with placeholders substituted at service start, the restart triggers fire on changes to the template, not to the secret, so a service can run indefinitely against a stale value while every unit looks healthy. Declare the dependency explicitly, and verify with two checks: the rendered runtime file shows the new value with no placeholders left, and the store’s copy of the template still shows only placeholders. (Lesson 5.9.)
17. The git index is shared process state. Two agents in one working tree share one index, so any commit in that tree commits whatever is staged at that moment regardless of who staged it. Give each agent its own worktree; that is the fix. Where one tree is shared anyway, name explicit paths, never stage a directory or everything, and accept that staging and committing in one invocation narrows the window without closing it. Never reset a shared tree hard, since in-flight work from other threads is the normal state there, and if your files land in someone else’s commit, verify they landed intact and add a commit that explains it rather than rewriting theirs. (Lesson 6.11.)
18. A partial mask is not a mask. When redacting a credential for a transcript, log or report, never print any part of the value, not a prefix, a suffix or a fixed-length slice; reason and report on names, classes and lengths only. Prove any automated redaction against the actual shape of the data it will meet, since a pattern written for a single-line value silently fails to match the same field across several lines and then prints it in full. (a lesson that is not published.)
19. A scratchpad is not storage. Anything written only into a session-scoped scratch directory does not survive a session boundary, a machine move or a context compaction, and a later rewrite from memory carries neither the reasoning nor the hard-won edge cases. Move anything reusable into the repository as soon as it works, keep it self-contained, never put a credential in a scratchpad even during testing, and check every scratchpad the prior session used before calling a migration complete. (No numbered lesson; a placement gap found in this theme.)
20. Disable a declaration, do not delete it. When something is turned off because it was measured, tried or superseded, keep the declaration in place, disabled, with the evidence that led to the decision and the condition that would reverse it recorded at the declaration itself; the expensive part was the measurement, not the configuration. Note that disabling a declaration does not stop what is already running, and delete only a genuine duplicate of an already-disabled block. (No numbered lesson; a placement gap found in this theme.)
21. A file’s modification time is not the time of the event it records. An on-disk timestamp reflects the most recent write to that path, which may be a checkout, a copy or a redeploy long after the event the contents describe. When timing matters, find the field that records the event itself, a captured-at value in the data, the job’s own log, the file’s history in version control, rather than trusting the filesystem’s bookkeeping. (Lesson 6.17.)
22. Rehearse on hardware that is allowed to fail. Before an agent operates a system it has not operated before, it builds the same thing in a sandbox it can break, records what the sandbox proved and what it could not, and treats the real hardware as a second experiment that re-checks the first. Skipping this turns every first attempt into a production incident; on Node0 the sandbox absorbed a failover assumption, four DHCP defects and a storage cluster’s first total power loss. (Lesson 6.18.)
23. Storage earns trust by time under load, not by a health check. A new drive gets a burn-in pass before it joins a pool, a new pool gets a full scrub before anything depends on it, and a new tier spends its first weeks behind parity that can lose a drive or with a second copy elsewhere, because storage fails early and more than anything else: on Node0, six drives and a controller in three weeks, including one at 297 hours. (Lesson 1.18.)
24. Batch and rank the work that needs hands. An agent iterates on configuration in seconds; every physical step costs a person’s time and usually a host outage around it, so an agent working a hardware fault exhausts what software can rule out first, then groups every request for hands into one visit, ordered by what each step would eliminate, with the parts and the read-back for each written down in advance. Iterating on a rack the way one iterates on a config turned one board diagnosis into a week. (Lesson 1.2.)
25. Nothing critical runs on a laptop that travels, and no secret lives on one outside a vault. A machine that closes and leaves the building sleeps, changes networks, sits outside the firewall half the time and can be lost, so it is a client, whatever it can do; every service the site depends on runs on hardware that stays, and a travelling laptop holds only a revocable way to ask for a secret, never a copy. A laptop racked as a permanent host is exempt, because it does not travel. (Lesson 6.19.)
26. Record the serial and the vendor of every incoming item, and test it on arrival. Second-hand above all: a used part is a claim waiting to be made, and the claim needs the serial, the seller and the test date written down before the part goes into service, because a defect found later against a vendor nobody recorded is a loss with nowhere to attribute it. One transfer switch that arrived with a mismatched controller cost about $550 that way. (Lesson 5.12.)
27. The same command name is not the same program on every machine. macOS, Windows and Linux hosts, and different releases of one tool, differ in flags, transports and output, and an agent carries a command learned on one platform to the next as if it were the same. A runbook that names a command names the operating system and version it was proven on, and a command is proven on a new platform before it is trusted there, with nothing destructive folded into the proof. A copy that failed quietly on one such difference is how three recordings were lost. (Lesson 3.1.)