Learn from what broke
A hundred and sixteen lessons from building Node0 between July and September 2026: a hundred and thirteen published from a collection of a hundred and seventeen, and three added on 20260922. Each one: what happened, what it cost, what changed, and the check that catches it now.
Collection v0.1, 20260920, from the runbooks, the inventory’s journal, the agents’ memories and the incident records. A hundred and thirteen of the hundred and seventeen are published, and three more, on the sandbox, on burning storage in, and on laptops that travel, were added on 20260922; the four that concern credentials are listed by shape only on their theme page. Sanitized 20260921: humans other than Christoph by role, agents by name, no addresses or serials.
The one sentence, before the hundred and sixteen. Almost every lesson below is the same lesson wearing different hardware. A check was skipped, or the check could not tell the outcomes apart, and the output looked like evidence. The counter-habit that worked, when it was used, is to state what result would falsify the claim, then go and get that result before writing the claim down.
Six themes #
In the order things happened inside each. Things that happened and changed nothing are listed as declined at the end of every theme, so you can see what was left out and why.
Start with these #
Ten that have a turn in them, chosen for a reader who has never seen this site.
The day four failures went unnoticed, and none of them paged anyone
On one day (20260906) two Ceph hosts had been hung for ten hours, four switches sat within a few degrees of their automatic thermal shutdown, a 12 TB storage disk had been dead for 38 hours, and a power event had left two machines dark. Every one of them was found by accident.
A dirty bay connector wears two different disguises
A drive in one bay of ceph8 showed rising SATA CRC errors and a link trained down to 3.0 Gb/s over four days. It was drained and pulled, and a proven-good spare fitted into the same bay stalled at zero throughput with a clean SMART report and a clean 6.0 Gb/s link. Compressed air through the bay's connector cleared both signatures.
The second storage tier nobody was writing to
Proxmox Backup Server, the disk-level backup layer for the virtualisation cluster's guests, was running with a datastore configured on the central storage server, but a live audit found the cluster's backup scheduler completely empty: no scheduled job for any guest, and exactly one manually triggered test backup from 20260904 in the datastore.
An NVMe controller can die and leave no evidence
An NVMe drive on ceph8 stopped answering writes and reported 'device not ready' to every reset. It survived a PCI remove-and-rescan and a warm reboot of the whole host, and came back only after a full cold power cycle. A week later the same failure on a drive in ceph2 went straight to a cold cycle and recovered identically. Both drives' own SMART counters then read as if nothing had happened.
Two labels, corrected three times, before anyone traced a cable
Which UPS feeds which rack PDU was recorded by intent when the PDUs were configured, found electrically crossed once the units' voltages diverged, corrected once at the rack to a mapping that was itself still wrong, and only settled by a cable-by-cable physical check.
Building a dead-man's switch that only proves itself by being killed
Between 20260906 and 20260911 the alerting path was deliberately placed so it could survive the failure of the thing it watches. The notification server went on a different host from the primary storage server, and a permanent heartbeat alert was set to page if it ever went quiet. A live test, stopping the metrics engine and timing the page, proved it worked.
Never delete in the same command as an unverified copy
An agent ran an scp whose remote path contained escaped quotes. Modern OpenSSH scp speaks SFTP, so the quotes were literal path characters and the transfer failed silently. An unconditional rm -f in the same command deleted the originals anyway.
An agent caused a hardware fault, and it is on the record
An agent found that one transfer switch's controller identified itself as a lower-voltage model, with a blank serial and an implausible manufacture date, and read both its input sources at roughly half the real line voltage. On Christoph's explicit go-ahead the agent changed the unit's configured line voltage from 120 to 230. The controller immediately declared both sources under-voltage, logged faults on both, and lost redundancy entirely.
103% of a UPS, and a nameplate rating that was wrong for months
The two UPS units feeding Node0's busiest rack draw, summed, close to the rating of a single one of them, so that rack cannot lose one UPS and keep protected power. Separately, the constant used for one unit's rating in that percentage was found on 20260917 to be the 208 V nameplate figure, not the 240 V figure the units actually run at, overstating every failover percentage by 6.7 percent for months.
Two agents, one git index
More than one AI agent works the same git working tree concurrently. While one agent, authoring as Rigger, was part-way through an unrelated commit, a second agent staged 17 files for its own change. Rigger's commit ran first and swept all 17 files into it, with no error or conflict, and the second agent's commit immediately after reported nothing added to commit.
One lesson, in full #
Every lesson has the same shape: five facts, then the story for someone who was not there.
Never delete in the same command as an unverified copy
- date
- 20260723
- what happened
- An agent ran an scp whose remote path contained escaped quotes. Modern OpenSSH scp speaks SFTP, so the quotes were literal path characters and the transfer failed silently. An unconditional rm -f in the same command deleted the originals anyway.
- what it cost
- Three irreplaceable voicemail recordings destroyed permanently. Only their transcripts survived, because transcription happened to have run first.
- what changed
- A standing rule with no exemptions: never put a deletion in the same command or statement as the transfer it depends on, because the transfer can fail silently and the delete runs anyway. Transfer, verify by hash or byte count that someone actually observes, then delete, as three separate steps.
- the check now
- Every subsequent deletion in the project is gated on a verified-copy check, a checksum or a count, before any rm runs. And a runbook that names a command names the operating system and version it was proven on.
An AI agent working through a large file-reorganisation project destroyed
three of Christoph’s voicemail recordings in a single command. The command was
meant to move files to a new location and clean up the source, but the move
used a remote path with quote characters that a modern scp treats literally
rather than expanding through a shell. The copy silently failed while looking
like it might have worked. The delete that followed in the same command ran
anyway, because nothing had checked whether the copy actually landed.
Every other transfer in the project used a script that verified a checksum before deleting. This one was a quick interactive command, exempted because it was small. That exemption was the entire failure. The fix was not a smarter script. It was a rule with no exceptions: a transfer, then an explicit verification step that someone or something actually looks at, then the delete, always as three separate observed actions, never folded together for convenience.
The command looked small, quick and obvious, and that is exactly why nobody
thought to double-check it. The mechanism is worth recognising on your own
systems. scp switched to SFTP as its transport, and SFTP does not run your
path through a remote shell, so quoting that used to be necessary is now a way
to name a file that does not exist. The failure is quiet. Only the deletion is
loud.
The second lesson underneath it is about the tools themselves. The same command name does not mean the same program, or the same version, on every machine an agent reaches, and an agent that learned a command on one host will use it on the next as if it were the same. On Node0 the Macs, the Windows hosts and the Linux fleet all answer ssh and all differ underneath: macOS ships a version of rsync from 2006 that aborts on a flag every Linux rsync accepts and prints a usage page that reads like help; a shell alias on one machine replaced the listing command with a different program and broke every script that parsed its output; the Windows hosts speak PowerShell, not a Unix shell at all; and the scp above changed its transport between two releases of the same tool. Old Linux releases that some hardware depends on carry the same risk, though the site has not met that case yet. So a runbook that names a command names the operating system and the version it was proven on, and an agent moving to a different platform proves the command there before it trusts it, which on a destructive command means the three separate steps above (rule 27).
sanitized: checklist v0.1, 20260921; recording descriptions removed; voice pass 20260921
Rules we now run by #
Twenty-seven fleet rules came out of the collection and were written into the operating rules the agents work under. Six rules to start with, five from that list and one straight from a lesson, each with the failure it prevents:
- After any hard hang, power off and on. A warm reset does not clear a CPU IERR or a wedged NVMe controller; a real power cycle does. And a clean health reading afterwards proves nothing, so write the event down.
- Identify a drive, board or card by its own serial, never by slot, bay or letter. Letters shuffle on a reboot; slots get reseated; the serial is the thing.
- Clean the bay before condemning the drive. One contaminated connector wears two disguises; a soak test on a known-good spare tells them apart.
- Build and confirm the producer before the rule. An alert written ahead of the thing it watches pages a human about your own unfinished work, and creating an inventory record for a host starts paging about it before it is built.
- Bound every wait loop, or make it die with its own session. An abandoned loop locked the only dashboard account out for a day by re-arming its own lockout.
- Never delete in the same command as the copy it depends on. Copy, verify, delete, as three observed steps. No exemption for small.
All twenty-seven , with the lesson each one came from.
How this was produced, and what is next #
How a lesson earns its place here. A lesson derived from a document Christoph has reviewed inherits that review, and its source line names the document. A lesson written from a ruling or a conversation names that instead, and Christoph reads it before it is published. The source line is the difference, and it is on every lesson.
Six agents read the runbooks, the inventory’s journal entries and hardware records, the agents’ own memories and the incident records, one theme each, in parallel, with one shared brief. A seventh did the assembly, the tables and the placements. A fresh-eyes read and an independent review followed, and every lesson that lived only in an agent’s memory, or nowhere, was placed in the runbook or on the device it belongs to. Six more agents derived the public pages from that collection, one theme each, through the sanitization checklist, and Christoph read them. The collection is dated. It grows by capture, not by edit.
Source: node0 lessons v0.1 (the collection of 20260920). Sanitized: checklist v0.1, 20260921; voice pass 20260921. Capture of 20260920; the private site keeps changing, this page does not. Part of oznog.com/node0.