Tok/s vs Agent Performance for Local LLMs, Explained
Tok/s shows how fast a model writes one reply. Agent jobs take many turns with growing prompts. Tom's Hardware agent times next to our Qwen3.8-27B estimates.
Updated · By RigPerDollar
As an Amazon Associate I earn from qualifying purchases. Some links on this page are affiliate links.
Tokens per second (tok/s) tells you how fast a local LLM writes one reply. Agent performance is how long it takes to finish a whole job, and an agent job is dozens of replies in a row. Each turn starts with the model reading a prompt that keeps getting longer. So a machine with a lower tok/s number can still finish agent work first.
Tom’s Hardware showed this in its RTX Spark review (October 15, 2026). It ran an AgentPerf-Local test with Qwen3.8-27B. The Surface Laptop Ultra with the RTX Spark chip finished in 18.9 minutes. A MacBook Pro with M5 Max took 24.74 minutes. A Ryzen AI Max+ 395 (Strix Halo) system took 42.1 minutes. Those are total task times in minutes. They include every turn, so they are a different measure from tok/s.
What each number measures
Our tok/s numbers are estimates for llama.cpp, the engine behind LM Studio and Ollama. They show decode speed: how fast the model writes new tokens (a token is about three quarters of a word). We fit them to tg128 runs, which means writing 128 tokens with a short prompt and one user. Our speed method has the formula and the benchmarks behind it.
An agent test times the full job. An agent is a model that works in a loop. It reads the task, calls a tool such as a file search or a test run, reads the result, and goes again. Each loop adds text to the prompt. The clock covers all of it:
- Prefill, the time spent reading the prompt before the first new token appears.
- Decode, the time spent writing the reply.
- Both of those again on every turn, with more context each time.
Artificial Analysis publishes an open method called AA-AgentPerf-Local. It replays recorded agent traces, so every machine gets the same turns and the same prompts. Artificial Analysis also runs its own leaderboard. Those are separate runs, and we don’t mix its times with the Tom’s Hardware numbers on this page.
The numbers side by side
Agent times are from Tom’s Hardware (October 15, 2026). Tok/s figures are our estimates for Qwen3.8-27B with its 17.44 GB Q4_K_M file, as listed on the Qwen3.8-27B page on October 22, 2026.
| Machine | Agent job time (Tom’s Hardware) | Our tok/s estimate |
|---|---|---|
| Surface Laptop Ultra (RTX Spark) | 18.9 min | No estimate yet (chip isn’t in our data) |
| MacBook Pro, M5 Max | 24.74 min | 21 (32-core GPU) to 28 (40-core GPU) |
| Ryzen AI Max+ 395 (Strix Halo) | 42.1 min | 11 |
The M5 Max comes with a 32-core or a 40-core GPU, so we show both. Two more machines for scale, from our data only (no agent time for them in this test):
| Machine | Our tok/s estimate | Estimate range |
|---|---|---|
| RTX 3090, 24 GB | 40 | 34 to 46 |
| DGX Spark, 128 GB | 13 | 11 to 15 |
| Ryzen AI Max+ 395, 128 GB | 11 | 9 to 13 |
The DGX Spark is a different NVIDIA product from the RTX Spark laptop chip. Our DGX Spark number uses constants fitted to NVIDIA graphics cards, so it may run high.
Where the two measures disagree
Compare the Mac and the Strix Halo. Our estimates say the M5 Max writes about 1.9 to 2.5 times as fast as the Strix Halo (21 or 28 tok/s against 11). In the Tom’s Hardware test, the Mac finished the agent job about 1.7 times as fast (24.74 minutes against 42.1). The order matches. The size of the gap does not.
The RTX Spark laptop finished about 2.2 times faster than the Strix Halo and about 1.3 times faster than the Mac. A tok/s estimate based on memory speed alone can’t tell you that, because finishing time depends on more than how fast the model writes.
Why a slower tok/s can finish first
- Prompt reading. Agent prompts get long: tool lists, file contents and every earlier step. Reading them leans on raw compute more than memory speed. Our estimate leaves this out, as the limits section explains.
- Long context slows writing too. The model reads its saved context (the KV cache) for every new token. Our methodology notes speeds often drop 20% or more by 8K tokens of context, and agent jobs go far past that.
- Speculative decoding. A small helper model guesses a few tokens ahead, and the big model checks them in one pass. When the guesses are right, output comes out faster. Our estimates don’t include it.
- Software. The runtime, backend and driver can each move a result by 10% or more. One machine may run a newer or better tuned stack.
- Power. A laptop or mini PC can run at a lower power limit, and a 40 minute job gives it time to heat up and slow down.
Fewer reasoning tokens can finish sooner
A model can finish sooner by writing fewer tokens, while each token still comes out at the same tok/s. The wait is the token count divided by the writing speed. A smaller count shortens the wait. The tok/s number stays put.
Thinklite-27B is Sparset’s fine-tune of Qwen3.8-27B. Its config names the same architecture as the Qwen3.8-27B config and has no mixture-of-experts layers. It is a dense model: every token runs the full network. Sparset says the tokenizer, chat template, and reasoning_effort setting are unchanged (model card; blog, November 2, 2026).
On Sparset’s own bf16 run, with reasoning_effort at xhigh, mean reasoning tokens on LiveCodeBench v6 were 27,589 for Qwen3.8-27B and 17,079 for Thinklite-27B (131 problems, one sample). Across 4,291 requests on six benchmarks, output tokens were 11,533,518 for Qwen3.8-27B and 5,822,599 for Thinklite-27B. Single-user decode was 26.2 tok/s for Qwen3.8-27B and 26.3 tok/s for Thinklite-27B. The blog calls the machine “our GPU” and does not name it. Its limits section says time-to-final-output was computed from each answer’s length and the measured single-user speed, rather than timed request by request. The about 2 times shorter wait for the whole suite is those two output totals at the same speed (the appendix divides every model by 26.2 tok/s). That is a token ratio, not a measured doubling of tok/s.
Sparset’s separate Q4_K_M llama.cpp card reports 69.46 and 70.09 tok/s for the two Q4 files, one user, on an unnamed GPU, and does not say which file is which. The speed-up it prints is an average of per-benchmark output-token ratios. That run’s LiveCodeBench reasoning means are 25,659 and 17,032, a different pair from the bf16 counts (GGUF card).
These are single vendor runs. We have not run Thinklite-27B. On November 5, 2026 its Hugging Face discussion list was empty, and a public web search did not find an independent accuracy or tok/s test.
Scores move on the bf16 run. At xhigh, IFEval was 89.1 for Thinklite-27B against 91.5 for Qwen3.8-27B, and MMLU-Pro was 84.4 against 86.3. The six-benchmark average was 89.82 against 90.17. At low effort the average was 84.70 against 86.88, and AIME 2025 was 81.7 against 90.0 (60 graded answers). The same table lists two other fine-tunes with a higher xhigh average, ThinkingCap-Qwen3.8-27B at 91.37 and Swift-1.5-Qwen3.8-27b at 91.12, with reasoning cuts of 45.8% and 35.6% against Thinklite’s 47.0%. That comparison is Sparset’s.
The LiveCodeBench counts are reasoning tokens only. They leave out the visible answer, prompt reading, and any speed change as context grows. The estimates in the tables above are something else again: llama.cpp writing 128 tokens at short context, one user, for Qwen3.8-27B’s 17.44 GB Q4_K_M file (how that estimate is defined). Dividing 27,589 or 17,079 by 40, 13, or 11 tok/s would mix the bf16 reasoning means with those short-prompt estimates. That quotient is not a measured request time on an RTX 3090, a DGX Spark, or a Strix Halo machine, and it is not a ranking of them. Sparset’s own computed median waits on LiveCodeBench are a different pair: 735.5 seconds for Qwen3.8-27B and 247.1 seconds for Thinklite-27B, from full output length at 26.2 tok/s plus about 0.1 seconds of prefill. This page leaves the counts as counts.
How to use tok/s for agent work
Tok/s is still useful. It ranks machines on the same software, and it tells you whether chat will feel fast. Memory matters even more for agents, because the context has to fit. On an RTX 3090 with 24 GB, Qwen3.8-27B fits with about 84K tokens of context at full precision, or about 159K with an 8-bit cache. The 128 GB DGX Spark and Strix Halo fit the model’s full 262,144 token window.
So start with fit and context, then read tok/s as a rough guide. If agents are your main use, look for an end-to-end agent test on the machine you want before you buy. You can check fit, context and speed for your own hardware on My rig.