Prompt reading vs writing speed: why one tok/s number isn't enough

Short answer

This guide explains why one tok/s figure isn’t enough. A local model first reads your prompt (time to first token, mostly limited by compute), then writes the reply (tok/s, mostly limited by memory bandwidth). High-compute machines read long prompts faster; high-bandwidth machines can write as fast or faster, and with several users or agents the big number is a total they share.

Two jobs: reading and writing

Every request to a local model has two parts.

Reading (prefill). The model reads the whole prompt before it can write anything. The wait you see is the time to first token. Reading handles many tokens at once, so it leans mostly on the chip’s raw compute.
Writing (decode). The model then writes the reply one token at a time (a token is about three quarters of a word). Each new token means reading the model’s weights from memory again, so writing speed leans mostly on memory bandwidth.

Most speed claims give one tok/s number, and that number is almost always writing speed. It says nothing about how long you wait before the first word appears.

For a short chat question the reading part is over quickly, so the writing speed is what you feel. Long prompts change that. If you paste in a codebase, or run an agent that carries its tool list, file contents and every earlier step in the prompt, the model may read tens of thousands of tokens before each reply. Then reading time can be most of the wait, and two machines with the same writing speed can feel very different.

Prompt caching: skipping the re-read

Runtimes can keep the work from reading a prompt and reuse it. This is called prompt caching, prefix caching or KV cache reuse. If the next request starts with the same text as an earlier one (the same system prompt, the same pasted files, the same chat so far), the runtime only reads the new part at the end.

vLLM calls this automatic prefix caching: it reuses the cached work for any new request that shares a prefix with an earlier one.
The llama.cpp server has a cache_prompt option that reuses the cache from a previous request, so the shared start of the prompt isn’t processed again, only the part that differs.

So in a chat, or an agent loop that only appends to its history, follow-up turns can start much sooner than the first one. The cache only helps while the start of the prompt stays the same. If an earlier part of the context changes (an edited file near the top, a reordered tool list, a trimmed history), everything after the change has to be read again. Caches also have limited room, and older entries get dropped.

Many users or agents: total vs per-user speed

Servers such as vLLM can work on several requests at once. With several users or agents, a headline number is often the total across all of them. Divide it by the number of users to get a rough per-user speed; one person never sees the whole total.

Each new user or agent also brings a prompt that has to be read. While the machine is reading a newcomer’s prompt, the others can slow down or wait, and the newcomer waits for their first token. So with many users, look for both the per-user writing speed and the time to first token, not only the total.

How to read a benchmark claim

Before you trust or compare a speed figure, check what it actually measured:

Model and exact quant or file. Not just “4-bit”: which file, from where.
Runtime and version. llama.cpp, vLLM, MLX or another engine, and which release or commit.
Prompt length. How many tokens the model read.
Output length. How many tokens it wrote.
Context. How much context was set and already in use.
Concurrency. How many users or requests at once, and whether the figure is a total or per user.
Speculative decoding. Whether a draft model or MTP was guessing tokens ahead.
Timing. What the clock covered: writing only, or reading plus writing.
Caching. Whether the prompt was already cached from an earlier run.
Power. For laptops: plugged in or on battery, and which power mode.
Date. When it was measured. Runtimes change quickly.

If a claim doesn’t say, treat that item as Unknown. Don’t compare two figures whose setups differ on these points, and don’t multiply a one-machine figure by the number of machines to guess what a cluster would do.

Examples

Third-party examples, not our estimates; conditions as stated, Unknown otherwise; numbers spoken in the video, on-screen charts not reviewed.

M5 Ultra vs two DGX Sparks

Alex Ziskind, video published 2026-10-01. An Apple M5 Ultra with 256 GB against two DGX Sparks running vLLM with the model split across both (TP=2). Model: DeepSeek V4 Flash, with a different 4-bit file on each platform. Runtime on the Mac and exact files: Unknown.

Writing, short prompt: about 38 tok/s on both.
Reading a 32K-token codebase: 50 s on the Mac, 17 s on the Sparks.
With 16K of context, the first ask took 21 s on the Mac and 8.3 s on the Sparks. The cached follow-up took 3 s and 1.4 s.
At 8 users, the Sparks’ 70 tok/s is a total shared by everyone, not a per-user speed.

The same writing speed with very different reading times is exactly what a single tok/s figure hides. The two platforms also ran different files, so this is not a like-for-like test.

MSI laptop with RTX Spark

Same reviewer, video published 2026-10-16. An MSI laptop with the RTX Spark chip running GPT-OSS-120B. File, runtime versions and power mode: Unknown.

llama.cpp: read prompts at about 1,800 tok/s and wrote at 65 tok/s for one user.
vLLM: read at 4,681 tok/s, wrote at 35 tok/s for one user, and 127.5 tok/s total across 8 users.

On the same machine, one runtime wrote faster for one user and the other read faster and served more users in total. Which one feels faster depends on your prompts and how many requests you send.

One DGX Spark with 8 coding agents

From the self-reported runs on our DGX Spark page. Qwen3.8-Flash-Next on one DGX Spark with NVIDIA Flare, 8 coding agents at once, measured Oct 28–29, 2026. Self-reported, not independently reproduced.

83.0 tok/s total across all 8 agents (mean of 5 runs), about 10 tok/s per agent (derived by us: 83.0 ÷ 8).
Median time to first token with 8 agents: 2.7 s.
The author’s reference engine, which serves one request at a time, showed a 63 s median time to first token on the same test, measured on a different date (Oct 22–24, 2026).

Full conditions are listed on the DGX Spark page.

What this means for our estimates

The tok/s figures on this site are writing-speed estimates for one user. They don’t include prompt reading time, prompt caching or multi-user totals. If your work involves long prompts, agents or several users at once, treat our number as one piece of the picture and look for measured reading times on the hardware you’re considering. Our methodology explains how the estimates are made and their limits.