What local AI models can the NVIDIA RTX A5000 run?
77 of our 115 models, up to Qwen3.6-35B-A3B with long context. It is the largest current model that keeps room for long context on the RTX A5000's 24 GB: at Q4 it fits with room for about 33K tokens of context (63K with 8-bit KV cache), at about 146 tok/s estimate. The largest that fits at all is K2-Horizon-MoVA-36B-A4B with room for only about 3K tokens of context (5K with 8-bit KV cache).
PriceNo price yet
Buy on AmazonEvery model, largest that fits first
| Model | Quant | File | Fits? | Speed estimate | Max context estimate |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | Q4_K_M | 22.29 GB | Fits | ~146 tok/s | 33K · 63K (8-bit) |
| Qwen3.5-35B-A3B | Q4_K_M | 22.29 GB | Fits | ~146 tok/s | 33K · 63K (8-bit) |
| NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB (2-bit) | native 2-bit | 22.02 GB | Fits | ~125 tok/s | 116K · 219K (8-bit) |
| MiniCPM-V 4.7 35B-A3B | Q4_K_M | 21.17 GB | Fits | ~148 tok/s | 87K · 164K (8-bit) |
| Nanbeige4.3-33B | Q4_K_M | 20.79 GB | Fits | ~31 tok/s | 10K · 19K (8-bit) |
| Xing4.0-29B-A4B | Q4_K_M | 20.06 GB | Fits | ~182 tok/s | 62K · 117K (8-bit) |
| Raon-1 mini preview Early support | Q4_K_M | 19.69 GB | Weights fit context unknown | ~118 tok/s | Unknown (no KV data) |
| PLaMo 3 Translate 31B Translation only | Q4_K_M | 19.53 GB | Fits | ~33 tok/s | 32K (full) |
| Meta Muse Glimmer 30B | Q4_K_M | 18.39 GB | Fits | ~35 tok/s | 128K (full) |
| Gemma 4 31B | Q4_K_M | 18.32 GB | Fits | ~35 tok/s | 40K · 90K (8-bit) |
| GLM-4.7-Flash | Q4_K_M | 18.31 GB | Fits | ~232 tok/s | 84K · 159K (8-bit) |
| Qwen3.6-27B | Q4_K_M | 17.98 GB | Fits | ~33 tok/s | 74K · 140K (8-bit) |
| Qwen3.5-27B | Q4_K_M | 17.98 GB | Fits | ~33 tok/s | 74K · 140K (8-bit) |
| IBM Granite 4.2 30B | Q4_K_M | 17.72 GB | Fits | ~36 tok/s | 19K · 37K (8-bit) |
| Microsoft Fara1.5 27B | Q4_K_M | 17.53 GB | Fits | ~34 tok/s | 81K · 153K (8-bit) |
| Qwen3.8-27B | Q4_K_M | 17.44 GB | Fits | ~34 tok/s | 82K · 155K (8-bit) |
| Gemma 4 26B A4B | Q4_K_M | 17.04 GB | Fits | ~185 tok/s | 256K (full) |
| Kvark Sparkle 1.1 27B | Q4_K_M | 16.81 GB | Fits | ~35 tok/s | 92K · 173K (8-bit) |
| LFM2.5-24B-A2B | Q4_K_M | 14.53 GB | Fits | ~170 tok/s | 128K (full) |
| Mistral-Small-3.2-24B-Instruct-2506 | Q4_K_M | 14.33 GB | Fits | ~45 tok/s | 51K · 97K (8-bit) |
| Devstral-Small-2505 | Q4_K_M | 14.33 GB | Fits | ~45 tok/s | 51K · 97K (8-bit) |
| gpt-oss-20b | Q4_K_M | 11.67 GB | Fits | ~214 tok/s | 128K (full) |
| phi-4 | Q4_K_M | 9.05 GB | Fits | ~69 tok/s | 16K (full) |
| JetBrains Mellum2.1 12B-A2.5B Thinking | Q4_K_M | 8.07 GB | Fits | ~241 tok/s | 128K (full) |
| Gemma 4 12B | Q4_K_M | 7.12 GB | Fits | ~85 tok/s | 256K (full) |
| NVIDIA Nemotron 3.5 Bolt E4B | Q4_K_M | 6.18 GB | Fits | ~97 tok/s | 64K (full) |
| Qwen3.5-9B | Q4_K_M | 6.17 GB | Fits | ~79 tok/s | 256K (full) |
| Ternary Bonsai 2 27B | PTQ1_0 ternary | 5.95 GB | Fits | ~81 tok/s | 254K · 256K (8-bit) |
| Microsoft Fara1.5 9B | Q4_K_M | 5.91 GB | Fits | ~81 tok/s | 256K (full) |
| Xiaomi MiMo-V2.6-Distill-Qwen-9B | Q4_K_M | 5.84 GB | Fits | ~82 tok/s | 256K (full) |
| ZoomGain-9B | Q4_K_M | 5.63 GB | Fits | ~84 tok/s | 256K (full) |
| K2-Horizon-7B | Q4_K_M | 5.59 GB | Fits | ~106 tok/s | 115K · 217K (8-bit) |
| webAI TwIL-8B Math and code reasoning | Q4_K_M | 5.35 GB | Fits | ~110 tok/s | 105K · 128K (8-bit) |
| IBM Granite 4.2 8B | Q4_K_M | 5.35 GB | Fits | ~110 tok/s | 105K · 128K (8-bit) |
| PLaMo 3 Translate 8B Translation only | Q4_K_M | 5.28 GB | Fits | ~111 tok/s | 32K (full) |
| LFM2.5-8B-A1B | Q4_K_M | 5.16 GB | Fits | ~183 tok/s | 128K (full) |
| Gemma 4 E4B | Q4_K_M | 4.98 GB | Fits | ~117 tok/s | 128K (full) |
| Spark-X2.6-4B | Q4_K_M | 3.54 GB | Fits | ~155 tok/s | 512K · 967K (8-bit) |
| Gemma 4 E2B | Q4_K_M | 3.46 GB | Fits | ~158 tok/s | 128K (full) |
| Qwen3.5-4B | Q4_K_M | 3.01 GB | Fits | ~124 tok/s | 256K (full) |
| NVIDIA Nemotron 3 Nano 4B | Q4_K_M | 2.9 GB | Fits | ~127 tok/s | 256K (full) |
| Microsoft Fara1.5 4B | Q4_K_M | 2.88 GB | Fits | ~127 tok/s | 256K (full) |
| ZoomGain-4B | Q4_K_M | 2.71 GB | Fits | ~131 tok/s | 256K (full) |
| IBM Granite 4.2 3B | Q4_K_M | 2.24 GB | Fits | ~222 tok/s | 128K (full) |
| Raon-1 nano Early support | Q4_K_M | 2.08 GB | Weights fit context unknown | ~150 tok/s | Unknown (no KV data) |
| Ternary Bonsai 2 9B | PTQ1_0 ternary | 2 GB | Fits | ~152 tok/s | 256K (full) |
| PLaMo 3 Translate 2B Translation only | Q4_K_M | 1.81 GB | Fits | ~258 tok/s | 32K (full) |
| LFM3-1.7B | Q4_K_M | 1.1 GB | Fits | ~191 tok/s | 256K (full) |
| LFM3-VL-2B | Q4_K_M | 1.1 GB | Fits | ~191 tok/s | 128K (full) |
| Ternary Bonsai 2 4B | PTQ1_0 ternary | 0.95 GB | Fits | ~200 tok/s | 256K (full) |
| MiniCPM-V 4.7 1B | Q4_K_M | 0.67 GB | Fits | ~218 tok/s | 256K (full) |
| PLaMo 3 Translate 0.8B Translation only | Q4_K_M | 0.57 GB | Fits | ~489 tok/s | 32K (full) |
| BotPsy-0.8B | Q4_K_M | 0.53 GB | Fits | ~228 tok/s | 256K (full) |
| Ternary Bonsai 2 2B | PTQ1_0 ternary | 0.43 GB | Fits | ~236 tok/s | 256K (full) |
| Qwen2.5-Coder-32B-Instruct Older | Q4_K_M | 19.85 GB | Fits | ~33 tok/s | 11K · 22K (8-bit) |
| DeepSeek-R1-Distill-Qwen-32B Older | Q4_K_M | 19.85 GB | Fits | ~33 tok/s | 11K · 22K (8-bit) |
| Qwen3-32B Older | Q4_K_M | 19.76 GB | Fits | ~33 tok/s | 12K · 22K (8-bit) |
| Qwen3-30B-A3B Older | Q4_K_M | 18.63 GB | Fits | ~214 tok/s | 40K (full) |
| Qwen3-Coder-30B-A3B-Instruct Older | Q4_K_M | 18.56 GB | Fits | ~214 tok/s | 44K · 83K (8-bit) |
| gemma-3-27b-it Older | Q4_K_M | 16.55 GB | Fits | ~39 tok/s | 69K · 128K (8-bit) |
| LFM2-24B-A2B Older | Q4_K_M | 14.42 GB | Fits | ~170 tok/s | 128K (full) |
| Qwen3-14B Older | Q4_K_M | 9 GB | Fits | ~69 tok/s | 40K (full) |
| Qwen2.5-Coder-14B-Instruct Older | Q4_K_M | 8.99 GB | Fits | ~69 tok/s | 32K (full) |
| DeepSeek-R1-Distill-Qwen-14B Older | Q4_K_M | 8.99 GB | Fits | ~69 tok/s | 69K · 128K (8-bit) |
| Mistral-Nemo-Instruct-2407 Older | Q4_K_M | 7.48 GB | Fits | ~82 tok/s | 92K · 128K (8-bit) |
| gemma-3-12b-it Older | Q4_K_M | 7.3 GB | Fits | ~83 tok/s | 128K (full) |
| NVIDIA-Nemotron-Nano-9B-v2 Older | Q4_K_M | 6.53 GB | Fits | ~75 tok/s | 128K (full) |
| Qwen3-8B Older | Q4_K_M | 5.03 GB | Fits | ~116 tok/s | 40K (full) |
| Llama-3.1-8B-Instruct Older | Q4_K_M | 4.92 GB | Fits | ~118 tok/s | 128K (full) |
| Qwen2.5-Coder-7B-Instruct Older | Q4_K_M | 4.68 GB | Fits | ~123 tok/s | 32K (full) |
| Mistral-7B-Instruct-v0.3 Older | Q4_K_M | 4.37 GB | Fits | ~130 tok/s | 32K (full) |
| Qwen3-4B Older | Q4_K_M | 2.5 GB | Fits | ~204 tok/s | 40K (full) |
| gemma-3-4b-it Older | Q4_K_M | 2.49 GB | Fits | ~205 tok/s | 128K (full) |
| Llama-3.2-3B-Instruct Older | Q4_K_M | 2.02 GB | Fits | ~239 tok/s | 128K (full) |
| Qwen3-0.6B Older | Q4_K_M | 0.48 GB | Fits | ~523 tok/s | 40K (full) |
| K2-Horizon-MoVA-36B-A4B | Q4_K_M | 22.37 GB | Fits | ~191 tok/s | 3K · 5K (8-bit) Tight |
| K2-Horizon-32B | Q4_K_M | 21.08 GB | Fits | ~31 tok/s | 7K · 13K (8-bit) Tight |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | Q4_K_M | 25.48 GB | No | — | — |
| Recursive Flint 40B-A2.5B | Q4_1 | 25.99 GB | No | — | — |
| Ripple-65B Preview | Q4_K_M | 38.8 GB | No | — | — |
| Llama-3.3-70B-Instruct Older | Q4_K_M | 42.52 GB | No | — | — |
| DeepSeek-R1-Distill-Llama-70B Older | Q4_K_M | 42.52 GB | No | — | — |
| Qwen2.5-72B-Instruct Older | Q4_K_M | 47.42 GB | No | — | — |
| Qwen3-Coder-Next | Q4_K_M | 48.53 GB | No | — | — |
| Aleph Alpha Kolibri-1 78B-A3.5B | Q4_K_M | 49.76 GB | No | — | — |
| NVIDIA Nemotron 3.5 Bolt 74B-A9B | UD-Q4_K_XL | 50.22 GB | No | — | — |
| gpt-oss-120b | Q4_K_M | 62.84 GB | No | — | — |
| NVIDIA Nemotron 3.5 Super VL 120B-A12B | Q4_K_M | 70.1 GB | No | — | — |
| Mistral Small 4 119B | Q4_K_M | 72.64 GB | No | — | — |
| GLM-4.5-Air Older | Q4_K_M | 73.5 GB | No | — | — |
| Qwen3.5-122B-A10B | Q4_K_M | 77.62 GB | No | — | — |
| Mistral Medium 3.5 128B | Q4_K_M | 78.41 GB | No | — | — |
| Ornith-1.5-122B | Q4_K_M | 78.92 GB | No | — | — |
| NVIDIA Nemotron 3 Super 120B-A12B Older | UD-Q4_K_M | 82.54 GB | No | — | — |
| Swift 1.5 Qwen3.8-Flash-Next | Q4_K_M | 119.6 GB | No | — | — |
| Qwen3.8-Flash-Next | Q4_K_M | 119.6 GB | No | — | — |
| StepFun Step 3.7 Flash | UD-Q4_K_M | 122.09 GB | No | — | — |
| Qwen3-235B-A22B | Q4_K_M | 142.15 GB | No | — | — |
| MiniMax-M3.1-Flash Unsloth 2-bit | UD-Q2_K_XL | 143.91 GB | No | — | — |
| DeepSeek-V4-Flash | UD-Q4_K_XL | 155.1 GB | No | — | — |
| MiniMax-M3.1-Flash Q2_K | Q2_K | 155.91 GB | No | — | — |
| Thinking Machines Inkling-Small Older | Q4_K_M | 159.34 GB | No | — | — |
| Xiaomi MiMo-V2.6-Flash 309B-A15B | MXFP4 | 167.37 GB | No | — | — |
| GLM-5.3-Flash | UD-Q4_K_XL | 199.71 GB | No | — | — |
| MiniMax-M3.1-Flash | Q4_K_M | 258.01 GB | No | — | — |
| Reflection Beam 1.0 501B-A23B | Q4_K_M | 302.41 GB | No | — | — |
| Thinking Machines Inkling 2 Small Early support | UD-Q4_K_XL | 354.15 GB | No | — | — |
| inclusionAI Ling-3.1-flash 560B-A25B | Q4_K_M | 357.97 GB | No | — | — |
| DeepSeek-V4.1-Flash (1017) | UD-Q4_K_XL | 500.71 GB | No | — | — |
| Meta Muse Spark Open 816B-A42B | UD-Q4_K_XL | 505.5 GB | No | — | — |
| Mistral Large 4 | UD-Q4_K_XL | 605 GB | No | — | — |
Q4_K_M unless the row says otherwise; UD- sizes are unsloth dynamic quants, the only Q4 build published for those models. Max context is an estimate of the longest context whose KV cache fits beside the model: default FP16 cache, then 8-bit (q8_0).
Speed calculator
estimateEstimate = bandwidth ÷ GB read per token (the model's listed GGUF file: Q4_K_M for most models, otherwise the quant named in the list; scaled by active/total parameters for MoE), scaled by a per-platform efficiency and fixed per-token costs calibrated to 45 measured llama.cpp benchmarks (2025–26). Fit = file size + 1.5 GB headroom ≤ total VRAM (on Macs, the part of unified memory macOS lets the GPU use by default), and room for at least 4K tokens of context with an 8-bit KV cache. For a model with no KV cache data, only the file and headroom are checked, so a fit says nothing about context. Multi-GPU: VRAM adds, speed stays one card's (a layer split runs the cards in turn); max context uses the total. Speed is for short context. Max context = what is left after the file and a 1 GB runtime reserve ÷ KV cache bytes per token, default FP16 and 8-bit (q8_0) cache. How speed works · how context works