What local AI models can the NVIDIA RTX 4000 SFF Ada Generation run?
61 of our 115 models, up to Meta Muse Glimmer 30B. That is the largest current model that fits the RTX 4000 SFF Ada Generation's 20 GB at Q4 with room for about 37K tokens of context (76K with 8-bit KV cache), at about 13 tok/s estimate.
Memory type: GDDR6 (source) · Bus width: 160-bit (source) · Released: 2023 (source)
- bandwidth_gbs: NVIDIA's March 2023 line card said 320 GB/s; the current datasheet and the 2025 line card say 280, used here.
Specs from first-party pages (NVIDIA), accessed 2026-10-18. Speeds use the NVIDIA (CUDA) constants (how).
As an Amazon Associate I earn from qualifying purchases.
Every model, largest that fits first
| Model | Quant | File | Fits? | Speed estimate | Max context estimate |
|---|---|---|---|---|---|
| Meta Muse Glimmer 30B | Q4_K_M | 18.39 GB | Fits | ~13 tok/s | 37K · 76K (8-bit) |
| GLM-4.7-Flash | Q4_K_M | 18.31 GB | Fits | ~113 tok/s | 12K · 23K (8-bit) |
| Qwen3.6-27B | Q4_K_M | 17.98 GB | Fits | ~13 tok/s | 15K · 28K (8-bit) |
| Qwen3.5-27B | Q4_K_M | 17.98 GB | Fits | ~13 tok/s | 15K · 28K (8-bit) |
| Microsoft Fara1.5 27B | Q4_K_M | 17.53 GB | Fits | ~13 tok/s | 21K · 41K (8-bit) |
| Qwen3.8-27B | Q4_K_M | 17.44 GB | Fits | ~13 tok/s | 23K · 43K (8-bit) |
| Gemma 4 26B A4B | Q4_K_M | 17.04 GB | Fits | ~84 tok/s | 78K · 160K (8-bit) |
| Kvark Sparkle 1.1 27B | Q4_K_M | 16.81 GB | Fits | ~14 tok/s | 32K · 61K (8-bit) |
| LFM2.5-24B-A2B | Q4_K_M | 14.53 GB | Fits | ~111 tok/s | 128K (full) |
| Mistral-Small-3.2-24B-Instruct-2506 | Q4_K_M | 14.33 GB | Fits | ~17 tok/s | 27K · 52K (8-bit) |
| Devstral-Small-2505 | Q4_K_M | 14.33 GB | Fits | ~17 tok/s | 27K · 52K (8-bit) |
| gpt-oss-20b | Q4_K_M | 11.67 GB | Fits | ~101 tok/s | 128K (full) |
| phi-4 | Q4_K_M | 9.05 GB | Fits | ~26 tok/s | 16K (full) |
| JetBrains Mellum2.1 12B-A2.5B Thinking | Q4_K_M | 8.07 GB | Fits | ~118 tok/s | 128K (full) |
| Gemma 4 12B | Q4_K_M | 7.12 GB | Fits | ~33 tok/s | 256K (full) |
| NVIDIA Nemotron 3.5 Bolt E4B | Q4_K_M | 6.18 GB | Fits | ~38 tok/s | 64K (full) |
| Qwen3.5-9B | Q4_K_M | 6.17 GB | Fits | ~35 tok/s | 256K (full) |
| Ternary Bonsai 2 27B | PTQ1_0 ternary | 5.95 GB | Fits | ~36 tok/s | 194K · 256K (8-bit) |
| Microsoft Fara1.5 9B | Q4_K_M | 5.91 GB | Fits | ~36 tok/s | 256K (full) |
| Xiaomi MiMo-V2.6-Distill-Qwen-9B | Q4_K_M | 5.84 GB | Fits | ~37 tok/s | 256K (full) |
| ZoomGain-9B | Q4_K_M | 5.63 GB | Fits | ~38 tok/s | 256K (full) |
| K2-Horizon-7B | Q4_K_M | 5.59 GB | Fits | ~42 tok/s | 88K · 167K (8-bit) |
| webAI TwIL-8B Math and code reasoning | Q4_K_M | 5.35 GB | Fits | ~44 tok/s | 81K · 128K (8-bit) |
| IBM Granite 4.2 8B | Q4_K_M | 5.35 GB | Fits | ~44 tok/s | 81K · 128K (8-bit) |
| PLaMo 3 Translate 8B Translation only | Q4_K_M | 5.28 GB | Fits | ~44 tok/s | 32K (full) |
| LFM2.5-8B-A1B | Q4_K_M | 5.16 GB | Fits | ~128 tok/s | 128K (full) |
| Gemma 4 E4B | Q4_K_M | 4.98 GB | Fits | ~47 tok/s | 128K (full) |
| Spark-X2.6-4B | Q4_K_M | 3.54 GB | Fits | ~64 tok/s | 406K · 767K (8-bit) |
| Gemma 4 E2B | Q4_K_M | 3.46 GB | Fits | ~66 tok/s | 128K (full) |
| Qwen3.5-4B | Q4_K_M | 3.01 GB | Fits | ~63 tok/s | 256K (full) |
| NVIDIA Nemotron 3 Nano 4B | Q4_K_M | 2.9 GB | Fits | ~65 tok/s | 256K (full) |
| Microsoft Fara1.5 4B | Q4_K_M | 2.88 GB | Fits | ~65 tok/s | 256K (full) |
| ZoomGain-4B | Q4_K_M | 2.71 GB | Fits | ~69 tok/s | 256K (full) |
| IBM Granite 4.2 3B | Q4_K_M | 2.24 GB | Fits | ~97 tok/s | 128K (full) |
| Raon-1 nano Early support | Q4_K_M | 2.08 GB | Weights fit context unknown | ~83 tok/s | Unknown (no KV data) |
| Ternary Bonsai 2 9B | PTQ1_0 ternary | 2 GB | Fits | ~85 tok/s | 256K (full) |
| PLaMo 3 Translate 2B Translation only | Q4_K_M | 1.81 GB | Fits | ~117 tok/s | 32K (full) |
| LFM3-1.7B | Q4_K_M | 1.1 GB | Fits | ~124 tok/s | 256K (full) |
| LFM3-VL-2B | Q4_K_M | 1.1 GB | Fits | ~124 tok/s | 128K (full) |
| Ternary Bonsai 2 4B | PTQ1_0 ternary | 0.95 GB | Fits | ~134 tok/s | 256K (full) |
| MiniCPM-V 4.7 1B | Q4_K_M | 0.67 GB | Fits | ~158 tok/s | 256K (full) |
| PLaMo 3 Translate 0.8B Translation only | Q4_K_M | 0.57 GB | Fits | ~285 tok/s | 32K (full) |
| BotPsy-0.8B | Q4_K_M | 0.53 GB | Fits | ~174 tok/s | 256K (full) |
| Ternary Bonsai 2 2B | PTQ1_0 ternary | 0.43 GB | Fits | ~187 tok/s | 256K (full) |
| gemma-3-27b-it Older | Q4_K_M | 16.55 GB | Fits | ~15 tok/s | 21K · 47K (8-bit) |
| LFM2-24B-A2B Older | Q4_K_M | 14.42 GB | Fits | ~111 tok/s | 128K (full) |
| Qwen3-14B Older | Q4_K_M | 9 GB | Fits | ~27 tok/s | 40K (full) |
| Qwen2.5-Coder-14B-Instruct Older | Q4_K_M | 8.99 GB | Fits | ~27 tok/s | 32K (full) |
| DeepSeek-R1-Distill-Qwen-14B Older | Q4_K_M | 8.99 GB | Fits | ~27 tok/s | 49K · 93K (8-bit) |
| Mistral-Nemo-Instruct-2407 Older | Q4_K_M | 7.48 GB | Fits | ~32 tok/s | 68K · 128K (8-bit) |
| gemma-3-12b-it Older | Q4_K_M | 7.3 GB | Fits | ~32 tok/s | 128K (full) |
| NVIDIA-Nemotron-Nano-9B-v2 Older | Q4_K_M | 6.53 GB | Fits | ~33 tok/s | 128K (full) |
| Qwen3-8B Older | Q4_K_M | 5.03 GB | Fits | ~46 tok/s | 40K (full) |
| Llama-3.1-8B-Instruct Older | Q4_K_M | 4.92 GB | Fits | ~47 tok/s | 104K · 128K (8-bit) |
| Qwen2.5-Coder-7B-Instruct Older | Q4_K_M | 4.68 GB | Fits | ~50 tok/s | 32K (full) |
| Mistral-7B-Instruct-v0.3 Older | Q4_K_M | 4.37 GB | Fits | ~53 tok/s | 32K (full) |
| Qwen3-4B Older | Q4_K_M | 2.5 GB | Fits | ~88 tok/s | 40K (full) |
| gemma-3-4b-it Older | Q4_K_M | 2.49 GB | Fits | ~88 tok/s | 128K (full) |
| Llama-3.2-3B-Instruct Older | Q4_K_M | 2.02 GB | Fits | ~106 tok/s | 128K (full) |
| Qwen3-0.6B Older | Q4_K_M | 0.48 GB | Fits | ~318 tok/s | 40K (full) |
| IBM Granite 4.2 30B | Q4_K_M | 17.72 GB | Fits | ~14 tok/s | 4K · 8K (8-bit) Tight |
| Qwen3-Coder-30B-A3B-Instruct Older | Q4_K_M | 18.56 GB | No | — | — |
| Gemma 4 31B | Q4_K_M | 18.32 GB | No | — | — |
| Qwen3-30B-A3B Older | Q4_K_M | 18.63 GB | No | — | — |
| PLaMo 3 Translate 31B Translation only | Q4_K_M | 19.53 GB | No | — | — |
| Raon-1 mini preview Early support | Q4_K_M | 19.69 GB | No | — | — |
| Qwen3-32B Older | Q4_K_M | 19.76 GB | No | — | — |
| Qwen2.5-Coder-32B-Instruct Older | Q4_K_M | 19.85 GB | No | — | — |
| DeepSeek-R1-Distill-Qwen-32B Older | Q4_K_M | 19.85 GB | No | — | — |
| Xing4.0-29B-A4B | Q4_K_M | 20.06 GB | No | — | — |
| Nanbeige4.3-33B | Q4_K_M | 20.79 GB | No | — | — |
| MiniCPM-V 4.7 35B-A3B | Q4_K_M | 21.17 GB | No | — | — |
| K2-Horizon-32B | Q4_K_M | 21.08 GB | No | — | — |
| NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB (2-bit) | native 2-bit | 22.02 GB | No | — | — |
| Qwen3.6-35B-A3B | Q4_K_M | 22.29 GB | No | — | — |
| Qwen3.5-35B-A3B | Q4_K_M | 22.29 GB | No | — | — |
| K2-Horizon-MoVA-36B-A4B | Q4_K_M | 22.37 GB | No | — | — |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | Q4_K_M | 25.48 GB | No | — | — |
| Recursive Flint 40B-A2.5B | Q4_1 | 25.99 GB | No | — | — |
| Ripple-65B Preview | Q4_K_M | 38.8 GB | No | — | — |
| Llama-3.3-70B-Instruct Older | Q4_K_M | 42.52 GB | No | — | — |
| DeepSeek-R1-Distill-Llama-70B Older | Q4_K_M | 42.52 GB | No | — | — |
| Qwen2.5-72B-Instruct Older | Q4_K_M | 47.42 GB | No | — | — |
| Qwen3-Coder-Next | Q4_K_M | 48.53 GB | No | — | — |
| Aleph Alpha Kolibri-1 78B-A3.5B | Q4_K_M | 49.76 GB | No | — | — |
| NVIDIA Nemotron 3.5 Bolt 74B-A9B | UD-Q4_K_XL | 50.22 GB | No | — | — |
| gpt-oss-120b | Q4_K_M | 62.84 GB | No | — | — |
| NVIDIA Nemotron 3.5 Super VL 120B-A12B | Q4_K_M | 70.1 GB | No | — | — |
| Mistral Small 4 119B | Q4_K_M | 72.64 GB | No | — | — |
| GLM-4.5-Air Older | Q4_K_M | 73.5 GB | No | — | — |
| Qwen3.5-122B-A10B | Q4_K_M | 77.62 GB | No | — | — |
| Mistral Medium 3.5 128B | Q4_K_M | 78.41 GB | No | — | — |
| Ornith-1.5-122B | Q4_K_M | 78.92 GB | No | — | — |
| NVIDIA Nemotron 3 Super 120B-A12B Older | UD-Q4_K_M | 82.54 GB | No | — | — |
| Swift 1.5 Qwen3.8-Flash-Next | Q4_K_M | 119.6 GB | No | — | — |
| Qwen3.8-Flash-Next | Q4_K_M | 119.6 GB | No | — | — |
| StepFun Step 3.7 Flash | UD-Q4_K_M | 122.09 GB | No | — | — |
| Qwen3-235B-A22B | Q4_K_M | 142.15 GB | No | — | — |
| MiniMax-M3.1-Flash Unsloth 2-bit | UD-Q2_K_XL | 143.91 GB | No | — | — |
| DeepSeek-V4-Flash | UD-Q4_K_XL | 155.1 GB | No | — | — |
| MiniMax-M3.1-Flash Q2_K | Q2_K | 155.91 GB | No | — | — |
| Thinking Machines Inkling-Small Older | Q4_K_M | 159.34 GB | No | — | — |
| Xiaomi MiMo-V2.6-Flash 309B-A15B | MXFP4 | 167.37 GB | No | — | — |
| GLM-5.3-Flash | UD-Q4_K_XL | 199.71 GB | No | — | — |
| MiniMax-M3.1-Flash | Q4_K_M | 258.01 GB | No | — | — |
| Reflection Beam 1.0 501B-A23B | Q4_K_M | 302.41 GB | No | — | — |
| Thinking Machines Inkling 2 Small Early support | UD-Q4_K_XL | 354.15 GB | No | — | — |
| inclusionAI Ling-3.1-flash 560B-A25B | Q4_K_M | 357.97 GB | No | — | — |
| DeepSeek-V4.1-Flash (1017) | UD-Q4_K_XL | 500.71 GB | No | — | — |
| Meta Muse Spark Open 816B-A42B | UD-Q4_K_XL | 505.5 GB | No | — | — |
| Mistral Large 4 | UD-Q4_K_XL | 605 GB | No | — | — |
Q4_K_M unless the row says otherwise; UD- sizes are unsloth dynamic quants, the only Q4 build published for those models. Max context is an estimate of the longest context whose KV cache fits beside the model: default FP16 cache, then 8-bit (q8_0).
Speed calculator
estimateEstimate = bandwidth ÷ GB read per token (the model's listed GGUF file: Q4_K_M for most models, otherwise the quant named in the list; scaled by active/total parameters for MoE), scaled by a per-platform efficiency and fixed per-token costs calibrated to 45 measured llama.cpp benchmarks (2025–26). Fit = file size + 1.5 GB headroom ≤ total VRAM (on Macs, the part of unified memory macOS lets the GPU use by default), and room for at least 4K tokens of context with an 8-bit KV cache. For a model with no KV cache data, only the file and headroom are checked, so a fit says nothing about context. Multi-GPU: VRAM adds, speed stays one card's (a layer split runs the cards in turn); max context uses the total. Speed is for short context. Max context = what is left after the file and a 1 GB runtime reserve ÷ KV cache bytes per token, default FP16 and 8-bit (q8_0) cache. How speed works · how context works