What local AI models can the NVIDIA A100 40GB PCIe run?
79 of our 115 models, up to Recursive Flint 40B-A2.5B. That is the largest current model that fits the A100 40GB PCIe's 40 GB at Q4_1 with room for its full 256K context, at about 346 tok/s estimate.
PriceNo price yet
Buy on AmazonEvery model, largest that fits first
| Model | Quant | File | Fits? | Speed estimate | Max context estimate |
|---|---|---|---|---|---|
| Recursive Flint 40B-A2.5B | Q4_1 | 25.99 GB | Fits | ~346 tok/s | 256K (full) |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | Q4_K_M | 25.48 GB | Fits | ~170 tok/s | 256K (full) |
| K2-Horizon-MoVA-36B-A4B | Q4_K_M | 22.37 GB | Fits | ~290 tok/s | 82K · 155K (8-bit) |
| Qwen3.6-35B-A3B | Q4_K_M | 22.29 GB | Fits | ~183 tok/s | 256K (full) |
| Qwen3.5-35B-A3B | Q4_K_M | 22.29 GB | Fits | ~183 tok/s | 256K (full) |
| NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB (2-bit) | native 2-bit | 22.02 GB | Fits | ~166 tok/s | 256K (full) |
| MiniCPM-V 4.7 35B-A3B | Q4_K_M | 21.17 GB | Fits | ~185 tok/s | 256K (full) |
| K2-Horizon-32B | Q4_K_M | 21.08 GB | Fits | ~60 tok/s | 66K · 125K (8-bit) |
| Nanbeige4.3-33B | Q4_K_M | 20.79 GB | Fits | ~61 tok/s | 83K · 157K (8-bit) |
| Xing4.0-29B-A4B | Q4_K_M | 20.06 GB | Fits | ~279 tok/s | 256K (full) |
| Raon-1 mini preview Early support | Q4_K_M | 19.69 GB | Weights fit context unknown | ~160 tok/s | Unknown (no KV data) |
| PLaMo 3 Translate 31B Translation only | Q4_K_M | 19.53 GB | Fits | ~65 tok/s | 32K (full) |
| Meta Muse Glimmer 30B | Q4_K_M | 18.39 GB | Fits | ~68 tok/s | 128K (full) |
| Gemma 4 31B | Q4_K_M | 18.32 GB | Fits | ~69 tok/s | 231K · 256K (8-bit) |
| GLM-4.7-Flash | Q4_K_M | 18.31 GB | Fits | ~335 tok/s | 198K (full) |
| Qwen3.6-27B | Q4_K_M | 17.98 GB | Fits | ~60 tok/s | 256K (full) |
| Qwen3.5-27B | Q4_K_M | 17.98 GB | Fits | ~60 tok/s | 256K (full) |
| IBM Granite 4.2 30B | Q4_K_M | 17.72 GB | Fits | ~71 tok/s | 79K · 128K (8-bit) |
| Microsoft Fara1.5 27B | Q4_K_M | 17.53 GB | Fits | ~61 tok/s | 256K (full) |
| Qwen3.8-27B | Q4_K_M | 17.44 GB | Fits | ~61 tok/s | 256K (full) |
| Gemma 4 26B A4B | Q4_K_M | 17.04 GB | Fits | ~283 tok/s | 256K (full) |
| Kvark Sparkle 1.1 27B | Q4_K_M | 16.81 GB | Fits | ~63 tok/s | 256K (full) |
| LFM2.5-24B-A2B | Q4_K_M | 14.53 GB | Fits | ~201 tok/s | 128K (full) |
| Mistral-Small-3.2-24B-Instruct-2506 | Q4_K_M | 14.33 GB | Fits | ~86 tok/s | 128K (full) |
| Devstral-Small-2505 | Q4_K_M | 14.33 GB | Fits | ~86 tok/s | 128K (full) |
| gpt-oss-20b | Q4_K_M | 11.67 GB | Fits | ~316 tok/s | 128K (full) |
| phi-4 | Q4_K_M | 9.05 GB | Fits | ~128 tok/s | 16K (full) |
| JetBrains Mellum2.1 12B-A2.5B Thinking | Q4_K_M | 8.07 GB | Fits | ~343 tok/s | 128K (full) |
| Gemma 4 12B | Q4_K_M | 7.12 GB | Fits | ~156 tok/s | 256K (full) |
| NVIDIA Nemotron 3.5 Bolt E4B | Q4_K_M | 6.18 GB | Fits | ~175 tok/s | 64K (full) |
| Qwen3.5-9B | Q4_K_M | 6.17 GB | Fits | ~123 tok/s | 256K (full) |
| Ternary Bonsai 2 27B | PTQ1_0 ternary | 5.95 GB | Fits | ~126 tok/s | 256K (full) |
| Microsoft Fara1.5 9B | Q4_K_M | 5.91 GB | Fits | ~126 tok/s | 256K (full) |
| Xiaomi MiMo-V2.6-Distill-Qwen-9B | Q4_K_M | 5.84 GB | Fits | ~127 tok/s | 256K (full) |
| ZoomGain-9B | Q4_K_M | 5.63 GB | Fits | ~130 tok/s | 256K (full) |
| K2-Horizon-7B | Q4_K_M | 5.59 GB | Fits | ~189 tok/s | 221K · 416K (8-bit) |
| webAI TwIL-8B Math and code reasoning | Q4_K_M | 5.35 GB | Fits | ~196 tok/s | 128K (full) |
| IBM Granite 4.2 8B | Q4_K_M | 5.35 GB | Fits | ~196 tok/s | 128K (full) |
| PLaMo 3 Translate 8B Translation only | Q4_K_M | 5.28 GB | Fits | ~198 tok/s | 32K (full) |
| LFM2.5-8B-A1B | Q4_K_M | 5.16 GB | Fits | ~210 tok/s | 128K (full) |
| Gemma 4 E4B | Q4_K_M | 4.98 GB | Fits | ~207 tok/s | 128K (full) |
| Spark-X2.6-4B | Q4_K_M | 3.54 GB | Fits | ~264 tok/s | 936K · 1M (8-bit) |
| Gemma 4 E2B | Q4_K_M | 3.46 GB | Fits | ~268 tok/s | 128K (full) |
| Qwen3.5-4B | Q4_K_M | 3.01 GB | Fits | ~172 tok/s | 256K (full) |
| NVIDIA Nemotron 3 Nano 4B | Q4_K_M | 2.9 GB | Fits | ~175 tok/s | 256K (full) |
| Microsoft Fara1.5 4B | Q4_K_M | 2.88 GB | Fits | ~175 tok/s | 256K (full) |
| ZoomGain-4B | Q4_K_M | 2.71 GB | Fits | ~179 tok/s | 256K (full) |
| IBM Granite 4.2 3B | Q4_K_M | 2.24 GB | Fits | ~352 tok/s | 128K (full) |
| Raon-1 nano Early support | Q4_K_M | 2.08 GB | Weights fit context unknown | ~195 tok/s | Unknown (no KV data) |
| Ternary Bonsai 2 9B | PTQ1_0 ternary | 2 GB | Fits | ~198 tok/s | 256K (full) |
| PLaMo 3 Translate 2B Translation only | Q4_K_M | 1.81 GB | Fits | ~396 tok/s | 32K (full) |
| LFM3-1.7B | Q4_K_M | 1.1 GB | Fits | ~227 tok/s | 256K (full) |
| LFM3-VL-2B | Q4_K_M | 1.1 GB | Fits | ~227 tok/s | 128K (full) |
| Ternary Bonsai 2 4B | PTQ1_0 ternary | 0.95 GB | Fits | ~233 tok/s | 256K (full) |
| MiniCPM-V 4.7 1B | Q4_K_M | 0.67 GB | Fits | ~245 tok/s | 256K (full) |
| PLaMo 3 Translate 0.8B Translation only | Q4_K_M | 0.57 GB | Fits | ~619 tok/s | 32K (full) |
| BotPsy-0.8B | Q4_K_M | 0.53 GB | Fits | ~251 tok/s | 256K (full) |
| Ternary Bonsai 2 2B | PTQ1_0 ternary | 0.43 GB | Fits | ~255 tok/s | 256K (full) |
| Qwen2.5-Coder-32B-Instruct Older | Q4_K_M | 19.85 GB | Fits | ~64 tok/s | 32K (full) |
| DeepSeek-R1-Distill-Qwen-32B Older | Q4_K_M | 19.85 GB | Fits | ~64 tok/s | 71K · 128K (8-bit) |
| Qwen3-32B Older | Q4_K_M | 19.76 GB | Fits | ~64 tok/s | 40K (full) |
| Qwen3-30B-A3B Older | Q4_K_M | 18.63 GB | Fits | ~315 tok/s | 40K (full) |
| Qwen3-Coder-30B-A3B-Instruct Older | Q4_K_M | 18.56 GB | Fits | ~316 tok/s | 203K · 256K (8-bit) |
| gemma-3-27b-it Older | Q4_K_M | 16.55 GB | Fits | ~75 tok/s | 128K (full) |
| LFM2-24B-A2B Older | Q4_K_M | 14.42 GB | Fits | ~201 tok/s | 128K (full) |
| Qwen3-14B Older | Q4_K_M | 9 GB | Fits | ~129 tok/s | 40K (full) |
| Qwen2.5-Coder-14B-Instruct Older | Q4_K_M | 8.99 GB | Fits | ~129 tok/s | 32K (full) |
| DeepSeek-R1-Distill-Qwen-14B Older | Q4_K_M | 8.99 GB | Fits | ~129 tok/s | 128K (full) |
| Mistral-Nemo-Instruct-2407 Older | Q4_K_M | 7.48 GB | Fits | ~150 tok/s | 128K (full) |
| gemma-3-12b-it Older | Q4_K_M | 7.3 GB | Fits | ~153 tok/s | 128K (full) |
| NVIDIA-Nemotron-Nano-9B-v2 Older | Q4_K_M | 6.53 GB | Fits | ~119 tok/s | 128K (full) |
| Qwen3-8B Older | Q4_K_M | 5.03 GB | Fits | ~205 tok/s | 40K (full) |
| Llama-3.1-8B-Instruct Older | Q4_K_M | 4.92 GB | Fits | ~209 tok/s | 128K (full) |
| Qwen2.5-Coder-7B-Instruct Older | Q4_K_M | 4.68 GB | Fits | ~216 tok/s | 32K (full) |
| Mistral-7B-Instruct-v0.3 Older | Q4_K_M | 4.37 GB | Fits | ~228 tok/s | 32K (full) |
| Qwen3-4B Older | Q4_K_M | 2.5 GB | Fits | ~330 tok/s | 40K (full) |
| gemma-3-4b-it Older | Q4_K_M | 2.49 GB | Fits | ~331 tok/s | 128K (full) |
| Llama-3.2-3B-Instruct Older | Q4_K_M | 2.02 GB | Fits | ~374 tok/s | 128K (full) |
| Qwen3-0.6B Older | Q4_K_M | 0.48 GB | Fits | ~645 tok/s | 40K (full) |
| Ripple-65B Preview | Q4_K_M | 38.8 GB | No | — | — |
| Llama-3.3-70B-Instruct Older | Q4_K_M | 42.52 GB | No | — | — |
| DeepSeek-R1-Distill-Llama-70B Older | Q4_K_M | 42.52 GB | No | — | — |
| Qwen2.5-72B-Instruct Older | Q4_K_M | 47.42 GB | No | — | — |
| Qwen3-Coder-Next | Q4_K_M | 48.53 GB | No | — | — |
| Aleph Alpha Kolibri-1 78B-A3.5B | Q4_K_M | 49.76 GB | No | — | — |
| NVIDIA Nemotron 3.5 Bolt 74B-A9B | UD-Q4_K_XL | 50.22 GB | No | — | — |
| gpt-oss-120b | Q4_K_M | 62.84 GB | No | — | — |
| NVIDIA Nemotron 3.5 Super VL 120B-A12B | Q4_K_M | 70.1 GB | No | — | — |
| Mistral Small 4 119B | Q4_K_M | 72.64 GB | No | — | — |
| GLM-4.5-Air Older | Q4_K_M | 73.5 GB | No | — | — |
| Qwen3.5-122B-A10B | Q4_K_M | 77.62 GB | No | — | — |
| Mistral Medium 3.5 128B | Q4_K_M | 78.41 GB | No | — | — |
| Ornith-1.5-122B | Q4_K_M | 78.92 GB | No | — | — |
| NVIDIA Nemotron 3 Super 120B-A12B Older | UD-Q4_K_M | 82.54 GB | No | — | — |
| Swift 1.5 Qwen3.8-Flash-Next | Q4_K_M | 119.6 GB | No | — | — |
| Qwen3.8-Flash-Next | Q4_K_M | 119.6 GB | No | — | — |
| StepFun Step 3.7 Flash | UD-Q4_K_M | 122.09 GB | No | — | — |
| Qwen3-235B-A22B | Q4_K_M | 142.15 GB | No | — | — |
| MiniMax-M3.1-Flash Unsloth 2-bit | UD-Q2_K_XL | 143.91 GB | No | — | — |
| DeepSeek-V4-Flash | UD-Q4_K_XL | 155.1 GB | No | — | — |
| MiniMax-M3.1-Flash Q2_K | Q2_K | 155.91 GB | No | — | — |
| Thinking Machines Inkling-Small Older | Q4_K_M | 159.34 GB | No | — | — |
| Xiaomi MiMo-V2.6-Flash 309B-A15B | MXFP4 | 167.37 GB | No | — | — |
| GLM-5.3-Flash | UD-Q4_K_XL | 199.71 GB | No | — | — |
| MiniMax-M3.1-Flash | Q4_K_M | 258.01 GB | No | — | — |
| Reflection Beam 1.0 501B-A23B | Q4_K_M | 302.41 GB | No | — | — |
| Thinking Machines Inkling 2 Small Early support | UD-Q4_K_XL | 354.15 GB | No | — | — |
| inclusionAI Ling-3.1-flash 560B-A25B | Q4_K_M | 357.97 GB | No | — | — |
| DeepSeek-V4.1-Flash (1017) | UD-Q4_K_XL | 500.71 GB | No | — | — |
| Meta Muse Spark Open 816B-A42B | UD-Q4_K_XL | 505.5 GB | No | — | — |
| Mistral Large 4 | UD-Q4_K_XL | 605 GB | No | — | — |
Q4_K_M unless the row says otherwise; UD- sizes are unsloth dynamic quants, the only Q4 build published for those models. Max context is an estimate of the longest context whose KV cache fits beside the model: default FP16 cache, then 8-bit (q8_0).
Speed calculator
estimateEstimate = bandwidth ÷ GB read per token (the model's listed GGUF file: Q4_K_M for most models, otherwise the quant named in the list; scaled by active/total parameters for MoE), scaled by a per-platform efficiency and fixed per-token costs calibrated to 45 measured llama.cpp benchmarks (2025–26). Fit = file size + 1.5 GB headroom ≤ total VRAM (on Macs, the part of unified memory macOS lets the GPU use by default), and room for at least 4K tokens of context with an 8-bit KV cache. For a model with no KV cache data, only the file and headroom are checked, so a fit says nothing about context. Multi-GPU: VRAM adds, speed stays one card's (a layer split runs the cards in turn); max context uses the total. Speed is for short context. Max context = what is left after the file and a 1 GB runtime reserve ÷ KV cache bytes per token, default FP16 and 8-bit (q8_0) cache. How speed works · how context works