BuildsGPUsModelsGuidesMethodologyNewsletter

Tokens per second calculator

Pick a GPU and a model; we estimate generation speed from memory bandwidth.

The formula: estimated tok/s = 0.76 × bandwidth (GB/s) ÷ GB read per token, where GB read per token is the Q4_K_M file size (scaled by active ÷ total parameters for mixture-of-experts models). A model fits when file size + 1.5 GB of runtime and short-context KV cache headroom is at most the total VRAM.

Worked example: Qwen3-32B is 19.76 GB (source) → 19.76 GB per token and ~21.3 GB of VRAM needed. On the NVIDIA GeForce RTX 3090 (936.2 GB/s): 0.76 × 936.2 ÷ 19.76 = ~36 tok/s estimate.

Mixture-of-experts: gpt-oss-120b is a 62.84 GB file but only 5.1B of 116.83B parameters are active, so it reads 2.74 GB per token while still needing ~64.3 GB of memory.

Last updated Oct 16, 2026. Speeds are bandwidth-based estimates, not benchmarks. Prices are dated snapshots and move daily; "check price" means we have no verified price.

Speed calculator estimate

Estimate = 0.76 × bandwidth ÷ GB read per token (Q4 file size, scaled by active/total parameters for MoE). Fit = file size + 1.5 GB headroom ≤ total VRAM. Multi-GPU: VRAM adds, speed uses one card's bandwidth (layer split). Short context only. How this works

The same formula, worked out

Every row is 0.76 × 936.2 GB/s ÷ GB read per token, for the NVIDIA GeForce RTX 3090 (24 GB, 936.2 GB/s). Only models that fit its memory are listed.

Model (Q4_K_M)FileNeeds (file + 1.5 GB)GB read per tokenEstimated tok/s estimate
Qwen2.5-Coder-32B-Instruct19.85 GB21.4 GB19.85 GB~36 tok/s
DeepSeek-R1-Distill-Qwen-32B19.85 GB21.4 GB19.85 GB~36 tok/s
Qwen3-32B19.76 GB21.3 GB19.76 GB~36 tok/s
Qwen3-30B-A3B18.63 GB20.1 GB2.01 GB (3.3B of 30.53B active)~353 tok/s
gemma-3-27b-it16.55 GB18.1 GB16.55 GB~43 tok/s
Mistral-Small-3.2-24B-Instruct-250614.33 GB15.8 GB14.33 GB~50 tok/s
Devstral-Small-250514.33 GB15.8 GB14.33 GB~50 tok/s
gpt-oss-20b11.67 GB13.2 GB2.01 GB (3.6B of 20.91B active)~354 tok/s

File sizes: Q4_K_M GGUFs from Hugging Face, linked on each model page, accessed 2026-10-15. The 0.76 factor and the 1.5 GB headroom allowance are explained in our methodology.

Questions people ask

How is tokens per second estimated?

Estimated tok/s = 0.76 x memory bandwidth in GB/s / GB read per token. GB read per token is the Q4_K_M file size, scaled by active divided by total parameters for mixture-of-experts models. 0.76 is the fraction of theoretical bandwidth we assume a real runtime reaches. Example: Qwen3-32B is dense, so it reads its whole 19.76 GB file per token, and on the NVIDIA GeForce RTX 3090 (936.2 GB/s) the estimate is 36 tok/s.

How much VRAM does a model need?

The Q4_K_M file size plus 1.5 GB for the runtime and a short-context KV cache. Qwen3-32B is 19.76 GB, so about 21.3 GB, which is why it fits a 24 GB card but not a 16 GB one. Long contexts need more than this allowance.

Does a second GPU double the speed?

No. VRAM adds up, so two 24 GB cards hold Llama-3.3-70B-Instruct (42.52 GB, about 44 GB with headroom), but a layer split reads one card at a time, so the estimate uses a single card's bandwidth: about 17 tok/s on two NVIDIA GeForce RTX 3090 cards.

Why are mixture-of-experts models estimated faster?

Only the active parameters are read per token. gpt-oss-120b is a 62.84 GB file with 5.1B of 116.83B parameters active, so the calculator uses 2.74 GB per token. You still need about 64.3 GB of memory to hold the whole file.

Related