Half a KVM stopped taking keystrokes, and the fix is a new KVM
- date
- 20260918, open as of 20260920
- what happened
- Eight of console1's sixteen ports, one entire physical bank, stopped forwarding USB keyboard input after a routine restart, while video switching kept working normally on every port. Three of the affected hosts had been logging USB errors for days beforehand. Two further restarts did not fix it.
- what it cost
- Eight hosts reduced to a view-only remote console, with no keyboard-based recovery path if any of them hangs.
- what changed
- Christoph's verdict is that this brand and model of KVM is not fit for the role and will be replaced when funds allow. Meanwhile any host that genuinely needs interactive console recovery is moved to a working port on the surviving bank, and the one spare port is kept as a manual bypass.
- the check now
- Type into a target and read its own input device to confirm a console is interactive, not merely watchable. A healthy picture proves nothing about keyboard delivery.
A remote console that still shows video is a console that will still be trusted, which is the whole problem here.
On 20260918, after a routine restart, one entire bank of Node0’s central KVM switch, eight of its sixteen ports, stopped delivering keyboard input to the machines behind it. Video switching kept working on every port, so the unit looked fine from the only angle most people check. Three of the affected hosts had been logging USB errors for days before the restart, which suggests the bank had been dying for a while and the restart finished it. Two more restarts changed nothing.
What that costs is specific: eight machines, three of them among the fleet’s largest compute servers, now have a console you can watch but cannot type into. The one situation a remote console exists for, a host that has hung before the network comes up and needs a key pressed, is exactly the situation this unit can no longer serve for those ports.
The verification technique is the transferable part, and it is what caught the fault: type into the target and read the input device on the other end, rather than trusting the screen. Picture and keystrokes travel different paths inside these units and either can fail alone.
The remedy for now is a workaround, while the replacement itself waits on funds. Christoph’s verdict is that this brand and model is not fit for the role and will be replaced when funds allow. Until then, hosts that genuinely need interactive recovery are moved to ports on the working bank, one spare port is held for a manual bypass by plugging the remote head directly into a target, and the rest are documented as having no keyboard-level recovery path. The fault is still open.
Source: node0 lessons v0.1, lesson 1.15. Sanitized: checklist v0.1, 20260921; affected host names; voice pass 20260921. Part of oznog.com/node0.
