Rung 4b: many kinds of models behind one gateway

What you get #

The compute box from rung 4 becomes the house’s utility box: small, private models, each a named route at the gateway every agent already uses (as built from 20261002):

• Search over your own files. Embeddings turn documents and queries into vectors, and a reranker puts the right document first: 95% of the time on the bench, against 80% for embeddings alone.
• Speech to text, without the audio leaving the house.
• Redaction and screening for your agents. A secret scanner and PII-Tracer clean what the site agent, Tender , is handed before it goes to a hosted model; injection screens are built and measured, and left off on purpose: the best combination caught 10 of 12 planted instructions with 6 false alarms in 15 clean texts, and labelled none of those that reached the agent, while the agent’s own refusal held in every run (below).
• One general text model on demand (Qwen3.6-35B-A3B) for drafts, digests and summaries a person checks, and Qwen3.8-27B loaded overnight for careful work such as the morning report.
• A supervisor that sheds the large models first under memory pressure, so search, transcription and redaction keep answering.

A hosted model sits behind the same gateway for work that benefits from a frontier model’s capability and speed and does not need to stay private. On the bench that is a coding plan, so only the coding route falls back to it; with a general plan, more routes could.

Each kind says what happens when the local box cannot answer. It falls back to a hosted model, or it fails closed (has no fallback). Private audio, documents and the agent’s redaction always fail closed: with redaction down, the agent’s text is withheld, not sent raw. With the local server held on 20260928, the fail-closed routes returned errors and the spend log showed no hosted request for them; the coding route answered from the hosted model, and the gateway metered it.

What still fails. Every model shares one GPU and one memory budget: the two large models do not fit together, so they take turns. This box is too slow to run an agent itself (rung 4 ).

Built first as four models at once (20260928). Before the utility tier, the box served the rung 4 chat model beside embeddings, the reranker and speech to text; the decisions and figures below from that day still hold for those three.

The utility tier (from 20261002) #

When the agent moved to a hosted model (rung 4 ), this Mac became the box for every model that is small, fast and private, so the agent’s data is cleaned here before it leaves. As the optional services arrive, their models join it here.

Rule: small models stay resident, one large model at a time loads on demand, and the supervisor unloads the large ones first. A box that stops its whole server under memory pressure loses every route at once; one that sheds its biggest model first keeps search, transcription and redaction answering.

• Memory, measured 20261002 against a 48 GiB ceiling. Everything resident plus the 35B-A3B idles at 35.6 to 36.9 GiB and peaked at 37.9 GiB under a realistic load, with pressure normal in all 29 samples and no swap. With the 27B in its place, about 43 GiB. The two large models together, about 65 GiB, do not fit, so they take turns.
• The unload order, tested under pressure. With 16 GiB held outside the server, the supervisor unloaded the 27B within 10 s; with 24 GiB, the 35B-A3B too. The server never stopped.
• The large models split the work. The 35B-A3B wrote a morning report in 9 s and got it wrong (resolved alerts listed as firing); the 27B took 85 s and got it mostly right. On a weekly digest both found 8 of 10 events, in 79 s against 463 s. So the 27B writes the morning report overnight and the 35B-A3B drafts digests and summaries that a person checks.
• The encoders need their own small service. llama.cpp serves the language models, but not classifiers like PII-Tracer; they run in one Python service behind the gateway (encoders.py ).

What each new model caught, on sets written before any model ran (20261002):

• No injection screen is a gate. The best combination missed 2 of 12 and raised 6 false alarms, and none labelled the instructions planted in the agent’s own intake. The screen is built and off; the agent’s own refusal is what held (the agents ).
• GLiNER waits for documents. Urgency is beyond it without training, so it is out of the agent’s path and comes back with a document service, where sorting by type is its job.

The server: llama.cpp, against oMLX, measured 20261002. llama.cpp’s router read prompts 17% faster, embedded 25% faster and reranked 18% faster, and its memory peaked at 33.1 GiB against oMLX’s 38.5 GiB, a peak 10.6 GiB above its idle, which is what counts against a ceiling. Their outputs matched (embeddings at a cosine of 0.9995). oMLX shed models by memory by itself, which is the feature a box of many models needs; our supervisor now does the same. Would change it: a later oMLX release with a steady peak; it was two days into 0.7.0.

Decisions and options #

The switcher: llama.cpp’s own router mode. One llama-server process loads each model on its first request and unloads the least recently used when a cap is reached, with per-model arguments from a presets file. The same binary serves chat, /v1/embeddings, /v1/rerank and /v1/audio/transcriptions. It adds nothing to download, and every model’s memory stays in one process tree the supervisor can add up. Would change it: your runtime lacks a router mode, or you mix runtimes. Then put llama-swap in front; it was our next candidate.

Only the router faces the network. The gateway reaches every model through the one router process, which listens on the network and requires the key. The process the router starts for each model listens only on the Mac itself (127.0.0.1), and the router does not pass the key on, so those processes have none. Nothing off the Mac can reach them; any account logged in on the Mac can, which is one more reason to keep the Mac dedicated.

Lower the supervisor’s free-memory threshold before loading four models. With all four working, free memory was 17 to 18% with pressure normal. Rung 4’s first threshold was 20%, and a test that left 19% free stopped the server. Ours is 5% from 20260928 (rung 4, and pins ).

Choose each model by a rule written before measuring. Ours, for the three new models (the chat model was chosen at rung 4, from Unsloth’s conversion, before this rule):

1. a permissive licence (Apache-2.0 or MIT);
2. GGUF files (llama.cpp’s model format) from the model’s authors or from ggml-org, never a third party’s conversion, pinned by revision and sha256;
3. runs on the pinned llama.cpp;
4. quantisation Q8_0 or better;
5. all four models loaded together fit the memory ceiling with at least 4 GiB to spare, measured at the working peak, not at idle;
6. then the best published score on the kind’s usual benchmark, from the model card;
7. a tie goes to the smaller model.

What the rule chose (model cards read 20260928):

Fallback or fail closed, per kind:

“Fails closed” means the route has no fallback and no hosted model shares its name. That held in the tested configuration: with the local server stopped on 20260928 these routes returned errors and the spend log showed no hosted request. A later routing change can undo it, so re-run that test after any change to the routes.

The hosted half: a subscription coding plan behind the gateway. We used z.ai’s GLM Coding Plan for GLM-5.3, from the same family as the local model. The gateway allows it 2 requests in flight and enforces the plan’s 60 requests an hour on the one key allowed to use it. Our plan covers coding tools. Putting it behind a gateway was Christoph’s decision, so read your own plan’s terms before you do the same. Since the local GLM model was retired on 20260929 (rung 4), the hosted model has no local fallback: a refusal from the plan now fails the request rather than quietly running locally.

Costs and measurements #

Measured 20260928 on the M1 Max (64 GB), llama.cpp b11146, through the gateway, except the hosted row. The test data was public (paragraphs of the node0 pages) or synthetic (speech made with macOS say). Full data: data/rung4 .

What cost us time #

• llama-server’s prompt cache defaults to 8 GiB of memory per model. With it on, the small models grew by about 0.5 GB per request until the supervisor stopped them at the ceiling, and reranking put the right document first 40% of the time instead of 95%. Set cache-ram = 0 for every model you do not chat with.
• LiteLLM’s openai/ transcription parser rejects llama.cpp’s usage object (input_tokens_details, where OpenAI’s is input_token_details). Use its mistral/ parser with the Mac as api_base; nothing goes to Mistral.

Runbooks #

Not yet written as runbooks. The bench’s own record is rung4b.md :

1. Write the model rule, read the model cards, choose, and pin every file by sha256. As built: rung4b.md and pins .
2. The presets file and the router’s cap on loaded models; the prompt cache off. As built: presets.ini .
3. One gateway route per kind; fallback or fail closed, stated in the configuration. As built: config.yaml .
4. The checks: every route answers; with the backend stopped, the fail-closed routes stay closed and the gateway’s spend log shows no hosted request; the coding route falls back and is metered. As built: the checks table in rung4b.md.
5. The measurements above, on your own public or synthetic data. As built: rung4b-bench.py .

Configuration #

Templates are not yet written. As built: the presets file presets.ini , the pins pins , and the gateway’s routes in config.yaml .

Before it: Rung 4: compute . After it: Rung 5: the edge and the second site . The whole ladder: the Seed .