Harness or Model?

When we tested the Seed’s site agent on a local model , Hermes on a 27B model on one GPU did less well than Claude Code on a hosted one. That left a question we could not answer: was it the model, or the harness around it? So this week we ran Hermes on the same hosted model as Claude Code, Sonnet 5.5, on the same faults, and graded both blind.

What we did #

Nous Research publishes a plugin, marked experimental, that lets Hermes use a Claude subscription through the Claude Code command-line tool, with Hermes keeping its own loop, tools and approvals. With it, Hermes ran three blocks of eight faults each, two with the power to fix things and one only watching. Then Claude Code ran two blocks with the power to fix, on the same faults in the same orders. Two graders, Codex and Rigger, graded every run blind against a rubric written before any of them ran, and sealed their grades before seeing each other’s. They agreed on 39 of 41 runs.

What we found #

An agent harness is the program around the model that runs its loop, tools and schedule. With the model the same, the two harnesses scored alike: Hermes 9 right of 16 faults, Claude Code 8, and neither did anything wrong or harmful. With the harness the same and only the model changed, Hermes went from 2 right to 5 on the same eight faults (from 4 when the local model had an extra rule to re-read its work), and it no longer ended turns in thought, one step short of a fix. In these tests, the model made the difference.

The samples are small. Claude Code itself got 5 right on these faults on 20261012 and 3 right on 20261023, so a gap of two is noise. Claude Code noticed faults sooner, but it also checks more often, every 10 minutes against Hermes’s 15. Hermes on Sonnet cost about $4 at list API prices for its three blocks, and wrote about a tenth of what it wrote on the local model. The agents page has the tables and the data has every grade.

What it means for you #

Choose the model first, then the harness that fits your site. That is a finding about routine site faults: restarts, drift, full disks, a lying health check. For other work, such as coding or long knowledge work, harnesses likely make a larger difference, and they always differ in how they schedule, limit and connect an agent. Many harnesses now run many models, so the harness stays a free choice. The answer may change as open models improve, or as larger models become practical to run at home. And the model is not everything: we compared two general-purpose harnesses, and research published this month, Harness² from Google Research and Northwestern University, found that a harness which reshapes itself for the task beats the same model in a default one. We have not tested that; the agents page says more.

Two things the test taught us apply to any site. When two agents share one account, give each model client its own home directory, or one can reach the other’s state. And watch every login your site depends on: our building agent’s login expired mid-test, and nothing said so until we looked, hours later. A check now tries each login every hour and warns well before a long-lived token expires.

The Seed’s agents are paused until the next work, which is a second brain for them: one place for the site’s notes and documents, which they can search and add to.