The number everyone quotes about self-hosted AI is the wrong one
July 26, 2026
If you have looked into running an AI model on your own machine, you have seen the pitch: buy the hardware once, stop paying per token forever. Every guide quotes the same metric to tell you whether your hardware is up to it — tokens per second, the speed at which words appear on screen.
For chat, that metric is fine. For a coding agent, it is close to irrelevant, and I spent a day measuring exactly why.
Two numbers, and only one of them gets quoted
Every request to a language model has two phases. First it reads your prompt. Then it writes a response. Those phases run at wildly different speeds, and the ratio between them decides whether local hosting is viable.
Benchmarks quote the writing speed, because it is the one you can see. It is also the one that barely matters for agent work, because of a detail specific to how coding agents operate: they resend an enormous prompt on every single turn.
I measured the agent I actually use. Its system prompt plus its tool definitions come to about 70 KB — roughly 18,000 tokens. Seventy percent of that is tool schemas: the machine-readable descriptions of the twenty-nine things the agent is allowed to do. All of it is re-read from scratch every turn, before a single word comes back.
So the question is not “how fast does it write?” It is “how fast does it read?” And on a CPU, those numbers are not close.
That 70-percent figure also points at the cheapest fix available, and it is not a hardware fix. If most of what you re-read every turn is descriptions of tools the model will not use on this turn, the answer is to stop sending them: a small router that reads the request and hands it to a narrow sub-agent carrying three relevant tools instead of twenty-nine. That is an architecture change, it costs nothing, and it cuts the dominant term. Before buying a machine to read a bloated prompt faster, it is worth asking why the prompt is that big.
The ceiling you can calculate yourself
Before trusting anyone’s benchmark, there is a sanity check worth knowing, because it rules out a surprising number of published claims.
Generating a token requires reading every weight in the model. So the fastest a machine can possibly generate is its memory bandwidth divided by the model’s size in memory. A machine with 614 GB/s of bandwidth running a 20 GB model cannot exceed about thirty tokens per second. Real-world throughput lands somewhere around 60 to 80 percent of that ceiling.
This is not a benchmark. It is arithmetic, and it is a hard limit — you cannot emit a token faster than you can read the weights once.
I ran this check against two AI-generated hardware guides someone sent me. Both quoted generation speeds above the ceiling for the hardware they described — one by a factor of four. A third claimed a 1.6-trillion-parameter model could be “squeezed onto” a 128 GB machine, which would require compressing each parameter to under two-thirds of one bit. Sub-one-bit quantization is not a thing.
Any of those numbers would have justified a five-figure hardware purchase. All three fail on a division you can do in your head.
What this looks like on hardware people actually own
I tested on a 2019 business ultrabook — a four-core, 15-watt mobile chip with integrated graphics and no CUDA. Not a workstation, but a very common shape of machine, and exactly the kind of thing people ask whether they can repurpose as an always-on agent host.
Running a well-regarded 4-billion-parameter model, quantized to about 3.4 GB:
| Phase | Measured |
|---|---|
| Generation | 5.15 tokens/sec |
| Prompt reading | 22.79 tokens/sec |
Generation at five tokens per second is slow but survivable. The second number is the one that ends the conversation. Eighteen thousand tokens at 22.79 per second is roughly thirteen minutes — just to read the prompt, before generating anything, on every turn. A five-step agent task becomes an hour.
I tried the obvious fix. Prompt caching, where the model reuses work from the previous turn, gave a genuine 3.1× speedup — reading rose from 25.8 to 80.9 tokens per second. Real, and not nearly enough. It brought thirteen minutes down to under four. Multiply by a multi-step task and it is still unusable.
Even stripping every tool from the agent — which is to say, making it not an agent — the remaining system prompt still takes about a minute per turn.
There is a second failure that shows up before the speed does. Small models are noticeably worse at calling tools correctly. In one test, a weak model made thirty malformed tool calls in the process of answering a question whose correct answer was a single word. It invoked a terminal with no command, an image analyzer with no image, a file writer with no path. It got there eventually. A frontier model answered the same question in twelve seconds with two clean calls.
What changes the arithmetic
There is an architecture worth understanding before writing local hosting off, because it inverts the ceiling calculation.
Most models are dense: every parameter participates in every token, so a 31-billion-parameter model reads all 31 billion parameters every time. Mixture-of-experts models are different. They hold every parameter in memory but activate only a fraction per token.
That means a mixture-of-experts model’s speed is governed by its active parameters, not its total. A 122-billion-parameter model with only 10 billion active can be several times faster than a 31-billion dense model, while being substantially more capable. For a machine with lots of memory but modest bandwidth — which describes every Apple silicon laptop — this is the whole ballgame, and most buying guides do not mention it.
It does not rescue the ultrabook. But it changes which models are worth considering on serious hardware, and it is the single most useful thing I learned.
Here is what actually fits in 128 GB, the maximum on a current high-end laptop:
| Model | Total / active | Approx. size | Fits? |
|---|---|---|---|
| Qwen3.5 122B (MoE) | 122B / 10B | ~61 GB | Yes — best speed-to-capability |
| Gemma 4 31B (dense) | 31B / 31B | ~20 GB | Yes |
| Llama 3.3 70B (dense) | 70B / 70B | ~35 GB | Yes, slowly |
| DeepSeek V4 Flash | 284B / 13B | ~158 GB | No — 30 GB over |
| Kimi K3 | 2.8T / 32B | ~350 GB | No — datacenter only |
The last two are worth flagging because both appear in guides recommending them for exactly this hardware. Neither fits, and the gap is not close.
One caveat on that table: those sizes are weights only, and weights are not the whole bill. Holding a conversation in memory costs memory too — every token in the context needs its attention state kept around. Modern models use compressed attention schemes that make this cheap at agent-sized contexts: a few gigabytes for eighteen thousand tokens. But it scales with context length and with how many requests you run at once. Push to a long context, or run two agents side by side, and it grows to tens of gigabytes — enough to push a model that “fits” on paper off the machine in practice. Whatever the table says, leave real headroom.
So what does heavy use actually cost?
Here is where it gets interesting, because the honest answer surprised me.
Take a genuinely heavy user: someone running a coding agent all day, every working day. Call it two hundred agent turns per day, twenty thousand input tokens per turn, fifteen hundred out. That is aggressive, sustained use.
Start from the published rates, since the whole argument rests on them. Everything is priced per million tokens, and every provider charges several times more for what the model writes than for what it reads:
| Model | Input | Output | Cached input |
|---|---|---|---|
| Claude Opus 5 | $5.00 | $25.00 | $0.50 |
| Claude Sonnet 5 | $3.00 | $15.00 | $0.30 |
| Claude Haiku 4.5 | $1.00 | $5.00 | $0.10 |
Prompt caching is the third column. Send the same prefix again within the cache window and the re-read costs a tenth of list. There is a surcharge on the turn that writes the cache, so treat these as slightly optimistic, not exact.
Run our heavy user through that. Two hundred turns at twenty thousand input tokens is four million input tokens a day; assume nine-tenths of it hits cache, since the agent’s system prompt and tool schemas are identical every turn. That is 3.6 million cached tokens at $0.50 and 400,000 fresh ones at $5.00 — $1.80 plus $2.00. Output is 300,000 tokens at $25.00, or $7.50. Total: $11.30 a day, and note where it went. Input is a third of the bill, and that is before the model has written a line.
| Model | Per day | Per month (22 days) |
|---|---|---|
| Claude Opus 5 | $11.30 | ~$249 |
| Claude Sonnet 5 | $6.78 | ~$149 |
| Gemini 3.1 Pro | $5.12 | ~$113 |
| Gemini 3.6 Flash | $3.39 | ~$75 |
Two things stand out.
First, caching changes the shape of the bill entirely. Once repeated input is nearly free, output tokens dominate — two-thirds of the Opus figure above is output. The anxiety people have about giant context windows is mostly misplaced; you pay for what the model writes, not what it reads twice.
Second, against a maxed-out 128 GB laptop at $7,349, a heavy Opus user pays that off in about two and a half years. That is not the absurd payback period I expected, and if you stopped reading here you might reasonably conclude the hardware wins.
The trap in that conclusion
Heavy use is exactly where local hosting looks best on a spreadsheet and worst in practice, and the reason is the number from the top of this post.
If prompt reading takes even 40 seconds per turn on good hardware — optimistic for an 18,000-token agent prompt on a laptop — then two hundred turns per day is over two hours of waiting, every day, before generation even starts. The heavier your usage, the more that compounds. You would be buying a machine to save $249 a month, and paying for it in hours.
The hardware that makes local viable for this workload is not a laptop. It is a GPU with an order of magnitude more bandwidth, and the people successfully running local coding agents are, without exception, running them on dedicated GPUs — not the thin-and-light machines the buying guides are pitched at.
The option this framing leaves out
I have been treating this as a binary — your laptop, or someone else’s API — and it is not. There is a middle option that fixes the actual problem: rent a dedicated GPU instance and run an open model on it yourself.
That gets you the thing the laptop cannot deliver. A datacenter accelerator has memory bandwidth several times a laptop’s, so prompt reading stops being the bottleneck, and the model weights and your code stay on a machine only you are using. No shared endpoint, no third-party retention question.
What it does not get you is a lower bill. A current-generation accelerator rents for a few dollars an hour, so eight hours a day across a working month runs comfortably north of what heavy Opus usage costs — and that is on-demand pricing for a machine that bills whether or not you are typing. Committed or interruptible capacity narrows the gap; it does not obviously close it. The economics only turn when the box is genuinely saturated, which usually means a team sharing it rather than one person.
So it is not a cost play either. It is the right answer to a different question — and for some people, that question is not optional.
When none of this is a choice
Everything above assumes you are free to send your code to an API and are just deciding whether to. Plenty of people are not.
Under stricter regulatory regimes — regulated industries, government work, anything with a formal data-residency or accreditation boundary — calling a public endpoint with proprietary source is simply off the table, whatever the pricing page says. In that world self-hosting is not an optimization anyone chose; it is the constraint the architecture starts from, and the conversation skips straight past laptops to how much dedicated capacity you need to make the latency survivable for a whole team.
Which is worth naming, because it explains a lot of enthusiastic local-hosting writing. Some of it comes from people who ran the numbers and disagree with me. A lot of it comes from people who never had the option, and whose advice does not transfer to anyone who does.
What I would actually tell someone
Do not buy hardware to save on API costs. At light usage the payback runs to decades. At heavy usage the payback looks reasonable but the latency makes the machine unusable for the thing you bought it for. There is no usage level where the trade is clean. Renting a real GPU fixes the latency, but it does not fix the cost — so it belongs in the privacy column, not the savings one.
Shrink the prompt before you upgrade the machine. Most of what my agent re-read every turn was descriptions of tools it was not about to use. Narrowing that is free, it helps on every backend including the paid ones, and it attacks the term that actually dominates.
Do run models locally for what they are good at. Bulk summarization, classification, transcription, image analysis — anything with a short prompt and no tool calling. Marginal cost genuinely zero, quality genuinely fine. That is a real win and it costs nothing to set up.
Do buy the machine if you want the machine. Privacy on sensitive documents, offline capability, not having a credit balance that can run dry mid-task — those are legitimate reasons, and they do not require the cost argument to hold. And if a compliance boundary makes the API a non-starter, none of this is a decision anyway; skip to sizing dedicated capacity. Just do not let a payback calculation be the thing that convinces you.
And check the arithmetic on anything you read, including this. Divide bandwidth by model size. If a quoted speed exceeds that number, the rest of the document is not worth your time — and in my sample of three, two of them failed.
Hardware and pricing figures current as of July 2026. Performance numbers are measured on my own machine, not vendor-supplied. Pricing from the Claude and Gemini published rates; model specifications from Hugging Face and DeepSeek.