The heartbeat that was not watching what mattered
- date
- found 20260917 to 20260918, closed 20260920
- what happened
- When an agent's operating session moved from a laptop to a dedicated in-fabric guest, a timer-based heartbeat was built to prove the guest was alive. Writing the operating runbook, rather than any outage, surfaced the gap. A timer proves the guest is up, not that the agent session inside it is running, attached, or doing anything.
- what it cost
- Nothing yet, because the gap was found by careful documentation rather than by the failure happening. The cost avoided was an indefinitely silent agent that everyone believed was being watched.
- what changed
- The heartbeat script now gates the beat on two local checks before it beats at all: a process named exactly claude, owned by the agent account, must exist, and the session's OAuth refresh token must not be expired. A third gate covers the second agent runtime's longer-lived token the same way.
- the check now
- A failed gate makes the heartbeat service itself report failed, which the existing dead-man alarm already pages on after three missed beats. Both new gates were exercised, by killing the process and by making the credential file unreadable, before being trusted.
This is a small, precise example of a mistake that is easy to make with any service running inside a monitored host. The health check that gets built first, because it is the easiest to build, answers whether the machine is real in some sense, not whether the thing you actually depend on is working inside it. A timer here fires on schedule regardless of whether the thing it was built to prove is actually running.
Two details in the fix are worth copying directly. The process check matches the process name exactly rather than by substring. A shell command that merely contains the word would otherwise satisfy it, which would turn the gate into a second thing that reports success for the wrong reason. And the credential check looks at the refresh token’s expiry, not the access token’s. The access token renews silently on use and proves nothing about an idle session. The refresh token is the actual login, and its expiry is the field that distinguishes a live session from an abandoned one.
The general rule fits in one sentence. A heartbeat that proves a virtual machine is alive is a different claim from a heartbeat that proves the process you care about is alive inside it. Make the timer refuse to beat unless it can prove the second claim, and prove the refusal works by causing it on purpose before you rely on it.
Source: node0 lessons v0.1, lesson 6.14. Sanitized: checklist v0.1, 20260921; hostname, credential file paths; voice pass 20260921. Part of oznog.com/node0.