An abandoned wait loop locked the only Grafana account out for a full day, by re-arming its own lockout

The mechanism here is counterintuitive enough to be worth sitting with. Something tripped the dashboard account’s failed-login lockout once. After that, every request from the abandoned readiness loop was rejected specifically because of the lockout. Each rejection counted as another failed attempt, which re-armed the same lockout. Twelve times a minute, for a day. The loop’s credentials were entirely correct, proven the moment the lockout finally cleared. Resetting the password would have changed nothing at all, because the lockout was a property of the account rather than of the secret being presented to it.

A background process that keeps retrying against a target which is actively rejecting it for exceeding a retry limit is not harmless just because it is authorised. It becomes the mechanism that keeps the door locked. That generalises to any automated retry against a rate-limited or lockout-protected target. The retry must eventually stop, or it converts a transient block into a self-sustaining one.

The loops themselves came from a pattern that is easy to reach for and easy to forget: an agent session waits for a service to come up, backgrounded on a remote host. Then the session ends and nothing ever kills the loop. Two of them were running, days apart, from work that had long since finished. The rule now is that such a wait is bounded by an explicit timeout, or it runs somewhere it will die with its session. A scan for lingering loop processes is part of finishing work on a host.

The second finding is the plainer one. The loop held its secret as a literal argument, so it sat in the process list, readable, for as long as the loop ran.