What local AI models can the NVIDIA GeForce RTX 4060 run?
8 GB GDDR6, 272 GB/s, 115 W TDP. Specs: TechPowerUp (accessed 2026-10-15). Price: check price
Buy on Amazon As an Amazon Associate I earn from qualifying purchases.
| Model (Q4) | File | Fits? | Speed estimate |
|---|---|---|---|
| Qwen3-0.6B | 0.48 GB | ✅ | ~431 tok/s |
| Llama-3.2-3B-Instruct | 2.02 GB | ✅ | ~102 tok/s |
| gemma-3-4b-it | 2.49 GB | ✅ | ~83 tok/s |
| Qwen3-4B | 2.5 GB | ✅ | ~83 tok/s |
| Mistral-7B-Instruct-v0.3 | 4.37 GB | ✅ | ~47 tok/s |
| Qwen2.5-Coder-7B-Instruct | 4.68 GB | ✅ | ~44 tok/s |
| Llama-3.1-8B-Instruct | 4.92 GB | ✅ | ~42 tok/s |
| Qwen3-8B | 5.03 GB | ✅ | ~41 tok/s |
| NVIDIA-Nemotron-Nano-9B-v2 | 6.53 GB | ❌ | — |
| gemma-3-12b-it | 7.3 GB | ❌ | — |
| Mistral-Nemo-Instruct-2407 | 7.48 GB | ❌ | — |
| Qwen2.5-Coder-14B-Instruct | 8.99 GB | ❌ | — |
| DeepSeek-R1-Distill-Qwen-14B | 8.99 GB | ❌ | — |
| Qwen3-14B | 9 GB | ❌ | — |
| phi-4 | 9.05 GB | ❌ | — |
| gpt-oss-20b | 11.67 GB | ❌ | — |
| Mistral-Small-3.2-24B-Instruct-2506 | 14.33 GB | ❌ | — |
| Devstral-Small-2505 | 14.33 GB | ❌ | — |
| gemma-3-27b-it | 16.55 GB | ❌ | — |
| Qwen3-Coder-30B-A3B-Instruct | 18.56 GB | ❌ | — |
| Qwen3-30B-A3B | 18.63 GB | ❌ | — |
| Qwen3-32B | 19.76 GB | ❌ | — |
| Qwen2.5-Coder-32B-Instruct | 19.85 GB | ❌ | — |
| DeepSeek-R1-Distill-Qwen-32B | 19.85 GB | ❌ | — |
| Llama-3.3-70B-Instruct | 42.52 GB | ❌ | — |
| DeepSeek-R1-Distill-Llama-70B | 42.52 GB | ❌ | — |
| Qwen2.5-72B-Instruct | 47.42 GB | ❌ | — |
| gpt-oss-120b | 62.84 GB | ❌ | — |
| GLM-4.5-Air | 73.5 GB | ❌ | — |
| Qwen3-235B-A22B | 142.15 GB | ❌ | — |
Speed calculator estimate
Estimate = 0.76 × bandwidth ÷ GB read per token (Q4 file size, scaled by active/total parameters for MoE). Fit = file size + 1.5 GB headroom ≤ total VRAM. Multi-GPU: VRAM adds, speed uses one card's bandwidth (layer split). Short context only. How this works