RigPerDollar

What GPU do you need to run Nanbeige4.3-33B?

A new Radeon RX 7900 XTX ($1,350). It is the cheapest priced graphics card that holds Nanbeige4.3-33B (about 22.3 GB at Q4) with room for about 10K tokens of context (19K with 8-bit KV cache), at about 36 tok/s estimate; that is the card price only, and it requires a separate compatible PC. 54 of the 157 GPUs and systems we track fit it.

Parameters33.12B
Q4_K_M GGUF20.79 GBsource
VRAM needed~22.3 GBfile + headroom
Single GPUs that fit54
ReleasedOct 24, 2026source
LicenseApache 2.0
Context window512K tokens

33.12B parameters (Hugging Face, accessed 2026-10-25). Q4_K_M GGUF: 20.79 GB (source). Architecture config (Hugging Face config.json): 52 attention layers × 8 KV heads × 128 head dim, so about 208 KB of FP16 KV cache per token of context (how we estimate context).

Dense LlamaForCausalLM, fine-tuned from Nanbeige/Nanbeige4.3-33B-Base; 33.12B is Hugging Face's safetensors total of 33,118,977,024 (BF16, about 66.26 GB). 512K context (max_position_embeddings 524288, rope_theta 1.2e8). Nanbeige and Unsloth publish no GGUF, so Q4_K_M is bartowski's (20,788,825,216 bytes); every bartowski size is in the table below. Nanbeige claims state-of-the-art agentic performance among open models around 30B (@nanbeige on X, Oct 25, 2026); that is their claim, not checked here. Release date is the Hugging Face repo creation date.

GGUF files and sizes

Exact sizes from Hugging Face, accessed 2026-10-25. The fit and context tables on this page use the Q4_K_M file (20.79 GB); a smaller file needs less memory by the difference. VRAM needed is the file + the same headroom.

FilePublished byBytesSizeVRAM neededSource
IQ2_XXSbartowski10,071,403,64810.07 GB~11.6 GBHugging Face
IQ2_XSbartowski10,634,407,04010.63 GB~12.1 GBHugging Face
IQ2_Sbartowski11,150,560,38411.15 GB~12.7 GBHugging Face
IQ2_Mbartowski12,187,167,87212.19 GB~13.7 GBHugging Face
Q2_Kbartowski13,528,816,76813.53 GB~15 GBHugging Face
IQ3_XXSbartowski14,489,959,55214.49 GB~16 GBHugging Face
Q3_K_Sbartowski15,211,261,05615.21 GB~16.7 GBHugging Face
IQ3_XSbartowski15,212,665,98415.21 GB~16.7 GBHugging Face
Q3_K_Mbartowski15,919,746,17615.92 GB~17.4 GBHugging Face
Q3_K_Lbartowski17,270,484,09617.27 GB~18.8 GBHugging Face
IQ3_Mbartowski17,802,005,63217.8 GB~19.3 GBHugging Face
IQ4_XSbartowski18,635,041,92018.64 GB~20.1 GBHugging Face
Q4_0bartowski19,348,057,21619.35 GB~20.9 GBHugging Face
Q4_K_Sbartowski19,740,519,55219.74 GB~21.2 GBHugging Face
Q4_K_Mbartowski20,788,825,21620.79 GB~22.3 GBHugging Face
IQ4_NLbartowski20,871,343,23220.87 GB~22.4 GBHugging Face
Q4_1bartowski21,200,497,79221.2 GB~22.7 GBHugging Face
Q4_K_Lbartowski22,344,395,90422.34 GB~23.8 GBHugging Face
Q5_K_Sbartowski23,664,311,42423.66 GB~25.2 GBHugging Face
Q5_K_Mbartowski25,073,368,19225.07 GB~26.6 GBHugging Face
Q6_K_Sbartowski27,750,272,12827.75 GB~29.3 GBHugging Face
Q6_Kbartowski29,016,857,72829.02 GB~30.5 GBHugging Face
Q6_K_Lbartowski30,404,324,48030.4 GB~31.9 GBHugging Face
Q8_0bartowski35,195,128,96035.2 GB~36.7 GBHugging Face

Other places to get it

Checked Nov 14, 2026. We list only copies we opened and checked. Check file hashes for copies from other uploaders.

Single GPUs that fit, cheapest first

Single GPU that fitsVRAMSpeed estimateMax context estimatePrice
AMD Radeon RX 7900 XTX24 GB~36 tok/s10K · 19K (8-bit)low $1,350avg $1,552 · new · source, 2026-11-02
NVIDIA GeForce RTX 309024 GB~38 tok/s10K · 19K (8-bit)low $1,499avg $1,551 · used · source, 2026-11-02
Apple Mac Studio M5 Max (32-core GPU, 36 GB)36 GB (~24.1 usable)~19 tok/s10K · 19K (8-bit)low $2,499new · source, 2026-10-16
NVIDIA GeForce RTX 409024 GB~41 tok/s10K · 19K (8-bit)low $2,910avg $3,100 · used · source, 2026-11-02
NVIDIA DGX Spark (64GB)64 GB~11 tok/s193K · 364K (8-bit)low $4,999new · source, 2026-11-13
Apple Mac Studio M5 Ultra (64-core GPU, 96 GB)96 GB (~72 usable)~47 tok/s230K · 433K (8-bit)low $5,499new · source, 2026-10-23
NVIDIA GeForce RTX 509032 GB~70 tok/s46K · 88K (8-bit)low $6,500avg $8,757 · new · source, 2026-11-02
Framework Desktop, AMD Ryzen AI Max+ PRO 495 (Strix Halo), 192 GB192 GB~10 tok/s512K (full)low $6,799new · source, 2026-10-16
NVIDIA DGX Spark (128GB)128 GB~11 tok/s486K · 512K (8-bit)low $6,949.99new · source, 2026-10-23
NVIDIA RTX A600048 GB~31 tok/s120K · 226K (8-bit)low $8,363new · source, 2026-11-02
Intel Arc Pro B6024 GB~13 tok/s10K · 19K (8-bit)No price yet
NVIDIA A1024 GB~25 tok/s10K · 19K (8-bit)No price yet
NVIDIA GeForce RTX 3090 Ti24 GB~41 tok/s10K · 19K (8-bit)No price yet
NVIDIA L424 GB~13 tok/s10K · 19K (8-bit)No price yet
NVIDIA Quadro RTX 600024 GB~28 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX 4500 Ada Generation24 GB~18 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX A500024 GB~31 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX A550024 GB~31 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX PRO 4000 Blackwell24 GB~28 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX PRO 4000 SFF Blackwell24 GB~18 tok/s10K · 19K (8-bit)No price yet
NVIDIA Tesla P4024 GB~11 tok/s10K · 19K (8-bit)No price yet
NVIDIA TITAN RTX24 GB~28 tok/s10K · 19K (8-bit)No price yet
AMD Radeon PRO V71028 GB~17 tok/s28K · 53K (8-bit)No price yet
AMD Radeon AI PRO R960032 GB~24 tok/s46K · 88K (8-bit)No price yet
AMD Radeon AI PRO R9600D32 GB~24 tok/s46K · 88K (8-bit)No price yet
AMD Radeon AI PRO R970032 GB~24 tok/s46K · 88K (8-bit)No price yet
AMD Radeon AI PRO R9700S32 GB~24 tok/s46K · 88K (8-bit)No price yet
AMD Radeon PRO V62032 GB~19 tok/s46K · 88K (8-bit)No price yet
AMD Radeon PRO W680032 GB~19 tok/s46K · 88K (8-bit)No price yet
AMD Radeon PRO W7800 32GB32 GB~21 tok/s46K · 88K (8-bit)No price yet
Intel Arc Pro B7032 GB~17 tok/s46K · 88K (8-bit)No price yet
NVIDIA Quadro GV10032 GB~35 tok/s46K · 88K (8-bit)No price yet
NVIDIA RTX 5000 Ada Generation32 GB~24 tok/s46K · 88K (8-bit)No price yet
NVIDIA RTX PRO 4500 Blackwell32 GB~36 tok/s46K · 88K (8-bit)No price yet
NVIDIA V100 PCIe 32GB32 GB~36 tok/s46K · 88K (8-bit)No price yet
NVIDIA V100S PCIe 32GB32 GB~45 tok/s46K · 88K (8-bit)No price yet
Apple M5 Pro 48GB48 GB (~36 usable)~13 tok/s65K · 122K (8-bit)No price yet
NVIDIA A100 40GB PCIe40 GB~61 tok/s83K · 157K (8-bit)No price yet
NVIDIA A800 40GB Active40 GB~61 tok/s83K · 157K (8-bit)No price yet
AMD Radeon PRO W7800 48GB48 GB~32 tok/s120K · 226K (8-bit)No price yet
AMD Radeon PRO W790048 GB~32 tok/s120K · 226K (8-bit)No price yet
AMD Radeon PRO W7900 Dual Slot48 GB~32 tok/s120K · 226K (8-bit)No price yet
Apple Mac Studio M5 Max (40-core GPU, 64 GB)64 GB (~48 usable)~25 tok/s120K · 226K (8-bit)No price yet
NVIDIA A4048 GB~28 tok/s120K · 226K (8-bit)No price yet
NVIDIA L4048 GB~35 tok/s120K · 226K (8-bit)No price yet
NVIDIA L40S48 GB~35 tok/s120K · 226K (8-bit)No price yet
NVIDIA Quadro RTX 800048 GB~28 tok/s120K · 226K (8-bit)No price yet
NVIDIA RTX 6000 Ada Generation48 GB~39 tok/s120K · 226K (8-bit)No price yet
NVIDIA RTX PRO 5000 Blackwell 48GB48 GB~53 tok/s120K · 226K (8-bit)No price yet
NVIDIA RTX PRO 5000 Blackwell 72GB72 GB~53 tok/s230K · 433K (8-bit)No price yet
NVIDIA A100 80GB PCIe80 GB~75 tok/s266K · 502K (8-bit)No price yet
NVIDIA RTX PRO 6000 Blackwell Max-Q96 GB~70 tok/s340K · 512K (8-bit)No price yet
NVIDIA RTX PRO 6000 Blackwell Server Edition96 GB~63 tok/s340K · 512K (8-bit)No price yet
NVIDIA RTX PRO 6000 Blackwell Workstation Edition96 GB~70 tok/s340K · 512K (8-bit)No price yet

Every device that runs it

Graphics cards, Macs and mini PCs from the wider hardware list behind My rig, smallest usable memory first. Not sure about yours? Check your exact rig.

Show all 96 devices that run it
DeviceTypeMemorySpeed estimateMax context estimatePrice
AMD Radeon RX 7900 XTXGPU24 GB~36 tok/s10K · 19K (8-bit)low $1,350avg $1,552 · new · source, 2026-11-02
Intel Arc Pro B60Workstation GPU24 GB~13 tok/s10K · 19K (8-bit)No price yet
NVIDIA A10Data center GPU24 GB~25 tok/s10K · 19K (8-bit)No price yet
NVIDIA GeForce RTX 3090GPU24 GB~38 tok/s10K · 19K (8-bit)low $1,499avg $1,551 · used · source, 2026-11-02
NVIDIA GeForce RTX 3090 TiGPU24 GB~41 tok/s10K · 19K (8-bit)No price yet
NVIDIA GeForce RTX 4090GPU24 GB~41 tok/s10K · 19K (8-bit)low $2,910avg $3,100 · used · source, 2026-11-02
NVIDIA L4Data center GPU24 GB~13 tok/s10K · 19K (8-bit)No price yet
NVIDIA Quadro RTX 6000Workstation GPU24 GB~28 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX 4500 Ada GenerationWorkstation GPU24 GB~18 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX A5000Workstation GPU24 GB~31 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX A5500Workstation GPU24 GB~31 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX PRO 4000 BlackwellWorkstation GPU24 GB~28 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX PRO 4000 SFF BlackwellWorkstation GPU24 GB~18 tok/s10K · 19K (8-bit)No price yet
NVIDIA RTX Spark N1X laptop (6144-core GPU, 24GB)Laptop24 GB~13 tok/s10K · 19K (8-bit)No price yet
NVIDIA Tesla P40Data center GPU24 GB~11 tok/s10K · 19K (8-bit)No price yet
NVIDIA TITAN RTXGPU24 GB~28 tok/s10K · 19K (8-bit)No price yet
Apple M3 Max (30-core GPU) 36GBMac36 GB (~24.1 usable)~13 tok/s10K · 19K (8-bit)No price yet
Apple M3 Pro 36GBMac36 GB (~24.1 usable)~6 tok/s10K · 19K (8-bit)No price yet
Apple M4 Max (32-core GPU) 36GBMac36 GB (~24.1 usable)~17 tok/s10K · 19K (8-bit)No price yet
Apple Mac Studio M5 Max (32-core GPU, 36 GB)Mac36 GB (~24.1 usable)~19 tok/s10K · 19K (8-bit)low $2,499new · source, 2026-10-16
AMD Radeon PRO V710Data center GPU28 GB~17 tok/s28K · 53K (8-bit)No price yet
AMD Radeon AI PRO R9600Workstation GPU32 GB~24 tok/s46K · 88K (8-bit)No price yet
AMD Radeon AI PRO R9600DWorkstation GPU32 GB~24 tok/s46K · 88K (8-bit)No price yet
AMD Radeon AI PRO R9700Workstation GPU32 GB~24 tok/s46K · 88K (8-bit)No price yet
AMD Radeon AI PRO R9700SWorkstation GPU32 GB~24 tok/s46K · 88K (8-bit)No price yet
AMD Radeon PRO V620Data center GPU32 GB~19 tok/s46K · 88K (8-bit)No price yet
AMD Radeon PRO W6800Workstation GPU32 GB~19 tok/s46K · 88K (8-bit)No price yet
AMD Radeon PRO W7800 32GBWorkstation GPU32 GB~21 tok/s46K · 88K (8-bit)No price yet
AMD Ryzen AI Max 385 (32GB)Mini PC32 GB~9 tok/s46K · 88K (8-bit)No price yet
AMD Ryzen AI Max 390 (32GB)Mini PC32 GB~9 tok/s46K · 88K (8-bit)No price yet
AMD Ryzen AI Max+ 395 (32GB)Mini PC32 GB~9 tok/s46K · 88K (8-bit)No price yet
Intel Arc Pro B70Workstation GPU32 GB~17 tok/s46K · 88K (8-bit)No price yet
NVIDIA GeForce RTX 5090GPU32 GB~70 tok/s46K · 88K (8-bit)low $6,500avg $8,757 · new · source, 2026-11-02
NVIDIA Quadro GV100Workstation GPU32 GB~35 tok/s46K · 88K (8-bit)No price yet
NVIDIA RTX 5000 Ada GenerationWorkstation GPU32 GB~24 tok/s46K · 88K (8-bit)No price yet
NVIDIA RTX PRO 4500 BlackwellWorkstation GPU32 GB~36 tok/s46K · 88K (8-bit)No price yet
NVIDIA RTX Spark N1X laptop (6144-core GPU, 32GB)Laptop32 GB~13 tok/s46K · 88K (8-bit)No price yet
NVIDIA V100 PCIe 32GBData center GPU32 GB~36 tok/s46K · 88K (8-bit)No price yet
NVIDIA V100S PCIe 32GBData center GPU32 GB~45 tok/s46K · 88K (8-bit)No price yet
Apple M3 Max (40-core GPU) 48GBMac48 GB (~36 usable)~17 tok/s65K · 122K (8-bit)No price yet
Apple M4 Max (40-core GPU) 48GBMac48 GB (~36 usable)~22 tok/s65K · 122K (8-bit)No price yet
Apple M4 Pro 48GBMac48 GB (~36 usable)~11 tok/s65K · 122K (8-bit)No price yet
Apple M5 Max (40-core GPU) 48GBMac48 GB (~36 usable)~25 tok/s65K · 122K (8-bit)No price yet
Apple M5 Pro 48GBMac48 GB (~36 usable)~13 tok/s65K · 122K (8-bit)No price yet
NVIDIA A100 40GB PCIeData center GPU40 GB~61 tok/s83K · 157K (8-bit)No price yet
NVIDIA A800 40GB ActiveWorkstation GPU40 GB~61 tok/s83K · 157K (8-bit)No price yet
AMD Radeon PRO W7800 48GBWorkstation GPU48 GB~32 tok/s120K · 226K (8-bit)No price yet
AMD Radeon PRO W7900Workstation GPU48 GB~32 tok/s120K · 226K (8-bit)No price yet
AMD Radeon PRO W7900 Dual SlotWorkstation GPU48 GB~32 tok/s120K · 226K (8-bit)No price yet
Apple M1 Max 64GBMac64 GB (~48 usable)~17 tok/s120K · 226K (8-bit)No price yet
Apple M1 Ultra 64GBMac64 GB (~48 usable)~32 tok/s120K · 226K (8-bit)No price yet
Apple M2 Max (30-core GPU) 64GBMac64 GB (~48 usable)~17 tok/s120K · 226K (8-bit)No price yet
Apple M2 Max (38-core GPU) 64GBMac64 GB (~48 usable)~17 tok/s120K · 226K (8-bit)No price yet
Apple M2 Ultra 64GBMac64 GB (~48 usable)~32 tok/s120K · 226K (8-bit)No price yet
Apple M3 Max (40-core GPU) 64GBMac64 GB (~48 usable)~17 tok/s120K · 226K (8-bit)No price yet
Apple M4 Max (40-core GPU) 64GBMac64 GB (~48 usable)~22 tok/s120K · 226K (8-bit)No price yet
Apple M4 Pro 64GBMac64 GB (~48 usable)~11 tok/s120K · 226K (8-bit)No price yet
Apple M5 Pro 64GBMac64 GB (~48 usable)~13 tok/s120K · 226K (8-bit)No price yet
Apple Mac Studio M5 Max (40-core GPU, 64 GB)Mac64 GB (~48 usable)~25 tok/s120K · 226K (8-bit)No price yet
NVIDIA A40Data center GPU48 GB~28 tok/s120K · 226K (8-bit)No price yet
NVIDIA L40Data center GPU48 GB~35 tok/s120K · 226K (8-bit)No price yet
NVIDIA L40SData center GPU48 GB~35 tok/s120K · 226K (8-bit)No price yet
NVIDIA Quadro RTX 8000Workstation GPU48 GB~28 tok/s120K · 226K (8-bit)No price yet
NVIDIA RTX 6000 Ada GenerationWorkstation GPU48 GB~39 tok/s120K · 226K (8-bit)No price yet
NVIDIA RTX A6000Workstation GPU48 GB~31 tok/s120K · 226K (8-bit)low $8,363new · source, 2026-11-02
NVIDIA RTX PRO 5000 Blackwell 48GBWorkstation GPU48 GB~53 tok/s120K · 226K (8-bit)No price yet
NVIDIA RTX Spark N1X laptop (6144-core GPU, 48GB)Laptop48 GB~13 tok/s120K · 226K (8-bit)No price yet
AMD Ryzen AI Max 385 (64GB)Mini PC64 GB~9 tok/s193K · 364K (8-bit)No price yet
AMD Ryzen AI Max 390 (64GB)Mini PC64 GB~9 tok/s193K · 364K (8-bit)No price yet
AMD Ryzen AI Max+ 395 (64GB)Mini PC64 GB~9 tok/s193K · 364K (8-bit)low $2,199.99new · source, 2026-11-13
NVIDIA DGX Spark (64GB)Mini PC64 GB~11 tok/s193K · 364K (8-bit)low $4,999new · source, 2026-11-13
NVIDIA RTX Spark N1X laptop (6144-core GPU, 64GB)Laptop64 GB~13 tok/s193K · 364K (8-bit)No price yet
Apple M2 Max (38-core GPU) 96GBMac96 GB (~72 usable)~17 tok/s230K · 433K (8-bit)No price yet
Apple M3 Max (30-core GPU) 96GBMac96 GB (~72 usable)~13 tok/s230K · 433K (8-bit)No price yet
Apple M3 Ultra 96GBMac96 GB (~72 usable)~33 tok/s230K · 433K (8-bit)No price yet
Apple Mac Studio M5 Ultra (64-core GPU, 96 GB)Mac96 GB (~72 usable)~47 tok/s230K · 433K (8-bit)low $5,499new · source, 2026-10-23
NVIDIA RTX PRO 5000 Blackwell 72GBWorkstation GPU72 GB~53 tok/s230K · 433K (8-bit)No price yet
NVIDIA A100 80GB PCIeData center GPU80 GB~75 tok/s266K · 502K (8-bit)No price yet
Apple M1 Ultra 128GBMac128 GB (~96 usable)~32 tok/s340K · 512K (8-bit)No price yet
Apple M2 Ultra 128GBMac128 GB (~96 usable)~32 tok/s340K · 512K (8-bit)No price yet
Apple M3 Max (40-core GPU) 128GBMac128 GB (~96 usable)~17 tok/s340K · 512K (8-bit)No price yet
Apple M4 Max (40-core GPU) 128GBMac128 GB (~96 usable)~22 tok/s340K · 512K (8-bit)No price yet
Apple M5 Max (40-core GPU) 128GBMac128 GB (~96 usable)~25 tok/s340K · 512K (8-bit)No price yet
NVIDIA RTX PRO 6000 Blackwell Max-QWorkstation GPU96 GB~70 tok/s340K · 512K (8-bit)No price yet
NVIDIA RTX PRO 6000 Blackwell Server EditionData center GPU96 GB~63 tok/s340K · 512K (8-bit)No price yet
NVIDIA RTX PRO 6000 Blackwell Workstation EditionWorkstation GPU96 GB~70 tok/s340K · 512K (8-bit)No price yet
AMD Ryzen AI Max 385 (128GB)Mini PC128 GB~9 tok/s486K · 512K (8-bit)No price yet
AMD Ryzen AI Max 390 (128GB)Mini PC128 GB~9 tok/s486K · 512K (8-bit)No price yet
AMD Ryzen AI Max+ 395 (128GB)Mini PC128 GB~9 tok/s486K · 512K (8-bit)low $3,499.99new · source, 2026-11-13
NVIDIA DGX Spark (128GB)Mini PC128 GB~11 tok/s486K · 512K (8-bit)low $6,949.99new · source, 2026-10-23
NVIDIA RTX Spark N1X laptop (6144-core GPU, 128GB)Laptop128 GB~13 tok/s486K · 512K (8-bit)No price yet
Apple M2 Ultra 192GBMac192 GB (~144 usable)~32 tok/s512K (full)No price yet
Apple M3 Ultra 256GBMac256 GB (~192 usable)~33 tok/s512K (full)No price yet
Apple M5 Ultra 256GBMac256 GB (~192 usable)~47 tok/s512K (full)No price yet
Framework Desktop, AMD Ryzen AI Max+ PRO 495 (Strix Halo), 192 GBMini PC192 GB~10 tok/s512K (full)low $6,799new · source, 2026-10-16
Apple M5 Ultra 512GBMac512 GB (~384 usable)~47 tok/s512K (full)No price yet

Multi-GPU: cards that need two or more

CardsTotal VRAMSpeed estimateMax context estimate
2 × Intel Arc B58024 GB~13 tok/s10K · 19K (8-bit)
2 × NVIDIA GeForce RTX 3060 12 GB24 GB~15 tok/s10K · 19K (8-bit)
2 × NVIDIA GeForce RTX 4070 SUPER24 GB~21 tok/s10K · 19K (8-bit)
3 × NVIDIA GeForce RTX 506024 GB~19 tok/s10K · 19K (8-bit)
3 × NVIDIA GeForce RTX 5060 Ti 8 GB24 GB~19 tok/s10K · 19K (8-bit)
2 × NVIDIA GeForce RTX 507024 GB~28 tok/s10K · 19K (8-bit)
2 × AMD Radeon RX 7800 XT32 GB~23 tok/s46K · 88K (8-bit)
2 × AMD Radeon RX 9060 XT 16 GB32 GB~12 tok/s46K · 88K (8-bit)
2 × AMD Radeon RX 907032 GB~24 tok/s46K · 88K (8-bit)
2 × AMD Radeon RX 9070 XT32 GB~24 tok/s46K · 88K (8-bit)
2 × NVIDIA GeForce RTX 5060 Ti 16 GB32 GB~19 tok/s46K · 88K (8-bit)
2 × NVIDIA GeForce RTX 5070 Ti32 GB~36 tok/s46K · 88K (8-bit)
2 × NVIDIA GeForce RTX 508032 GB~39 tok/s46K · 88K (8-bit)

Speed calculator

estimate

Estimate = bandwidth ÷ GB read per token (the model's listed GGUF file: Q4_K_M for most models, otherwise the quant named in the list; scaled by active/total parameters for MoE), scaled by a per-platform efficiency and fixed per-token costs calibrated to 45 measured llama.cpp benchmarks (2025–26). Fit = file size + 1.5 GB headroom ≤ total VRAM (on Macs, the part of unified memory macOS lets the GPU use by default), and room for at least 4K tokens of context with an 8-bit KV cache. For a model with no KV cache data, only the file and headroom are checked, so a fit says nothing about context. Multi-GPU: VRAM adds, speed stays one card's (a layer split runs the cards in turn); max context uses the total. Speed is for short context. Max context = what is left after the file and a 1 GB runtime reserve ÷ KV cache bytes per token, default FP16 and 8-bit (q8_0) cache. How speed works · how context works

Related