Best local LLM for 12GB VRAM: what fits and how fast
Which local LLMs fit a 12GB VRAM card like the RTX 3060 12GB, how much context each one gets, their estimated speeds, and when 12 GB stops being enough.
Updated · By RigPerDollar
As an Amazon Associate I earn from qualifying purchases. Some links on this page are affiliate links.
The best local LLM for 12GB VRAM depends on the job. The largest model file that fits is Microsoft phi-4. Of the bigger models, Google Gemma 4 12B holds the most context, and JetBrains Mellum2.1 is the fastest, at an estimated 144 tok/s on a used RTX 3060 12GB.
VRAM is the memory on your graphics card. The model file and its working memory both have to fit there. Context is how much text the model can hold at once, counted in tokens (pieces of words). Tok/s means tokens per second, or how fast it writes. Every speed in this guide is a RigPerDollar estimate for llama.cpp (a popular open-source program for running models locally) at short context, with a target of about 15% either way. We rank models by fit, context and estimated speed, and cards by estimated speed per dollar. We don’t test how good a model’s answers are.
Models that fit a 12 GB card
File sizes below are Q4 GGUF files, except Ternary Bonsai 2 27B, which ships as a 1.75-bit file. Q4 is a compressed format that shrinks a model so it fits in less memory. By our fit rule, a model fits when its file plus 1.5 GB of headroom fits in VRAM, and at least 4K tokens of context with an 8-bit cache still fit after that. Context numbers assume a 1 GB runtime reserve.
Each context cell shows two numbers. The first uses the default FP16 cache. The second uses an 8-bit (q8_0) cache, which stores the model’s working memory in a smaller format, so more context fits.
| Model | Q4 file | Context on 12 GB (FP16 / 8-bit) | Est. tok/s, RTX 3060 12GB | Est. tok/s, RTX 5070 |
|---|---|---|---|---|
| Microsoft phi-4 | 9.05 GB | 9K / 16K | 34 | 61 |
| JetBrains Mellum2.1 12B-A2.5B Thinking | 8.07 GB | full 128K / full 128K | 144 | 222 |
| Google Gemma 4 12B | 7.12 GB | 201K / 256K | 42 | 76 |
| Qwen3.5-9B | 6.17 GB | 143K / 256K | 43 | 71 |
| Ternary Bonsai 2 27B | 5.95 GB | 75K / 141K | 45 | 73 |
| Xiaomi MiMo-V2.6-Distill-Qwen-9B | 5.84 GB | 153K / 256K | 45 | 74 |
| ZoomGain-9B | 5.63 GB | 160K / 256K | 47 | 76 |
| K2-Horizon-7B | 5.59 GB | 35K / 67K | 53 | 94 |
| IBM Granite 4.2 8B | 5.35 GB | 33K / 63K | 55 | 98 |
| LFM2.5-8B-A1B | 5.16 GB | full 128K / full 128K | 143 | 177 |
| Gemma 4 E4B | 4.98 GB | full 128K / full 128K | 59 | 104 |
Some notes on the list:
- phi-4 (14.66B parameters) is the biggest file that fits. With an 8-bit cache it reaches 16K, which is its full window. With the default cache you get 9K.
- Mellum2.1 and LFM2.5-8B-A1B are mixture-of-experts (MoE) models. An MoE model has many parts, and only a few of them run for each token. Mellum2.1 has 12.15B parameters in total but only 2.5B active, which is why its estimate is so high. Its model card calls it a thinking model for agentic coding work, coding, math and reasoning.
- Ternary Bonsai 2 27B is PrismML’s ternary version of Qwen3.8-27B at 1.75 bits per weight. It needs PrismML’s own fork of llama.cpp, because stock llama.cpp rejects these files.
- MiMo-V2.6-Distill-Qwen-9B is a fine-tune of Qwen3.5-9B.
- ZoomGain-9B is inclusionAI’s vision-language model, trained from Qwen3.5-9B for fine-grained image detail. Its fit leaves out a 0.92 GB vision file. Its model page notes that Hugging Face states no license for its weights.
- Microsoft Fara1.5 9B (5.91 GB) also fits. It is a computer-use agent for web browsers, so we left it out of the picks.
Smaller models fit with full or near-full context too: Gemma 4 E2B (3.46 GB), Qwen3.5-4B (3.01 GB), NVIDIA Nemotron 3 Nano 4B (2.9 GB), Granite 4.2 3B (2.24 GB) and MiniCPM-V 4.7 1B (0.67 GB). KRAFTON’s Raon-1 nano (2.08 GB) fits on its weights, but only KRAFTON’s own llama.cpp fork can load it for now, and its context is unknown because we have no KV cache data for it. Older models that the site now marks as superseded still fit as well, including Qwen3-14B (9.0 GB, 11K / 22K), DeepSeek-R1-Distill-Qwen-14B and Qwen2.5-Coder-14B (both 8.99 GB, 9K / 18K), Mistral Nemo (7.48 GB) and Gemma 3 12B (7.3 GB).
Why context varies at similar file sizes
phi-4 and Gemma 4 12B have files of similar size, yet Gemma gets 201K tokens of context and phi-4 gets 9K. The difference is the KV cache.
As a model reads, it keeps notes on every token so it can refer back to them. Those notes are the KV cache, and they grow with every token of context. How many bytes each token costs depends on the model’s design. These figures come from each model’s config.json, at FP16:
| Model | KV cache per token | Context at FP16 on 12 GB |
|---|---|---|
| phi-4 | 204,800 bytes | 9K |
| Qwen3.5-9B | 32,768 bytes | 143K |
| Gemma 4 12B | 16,384 bytes (global layers) | 201K |
| Mellum2.1 | 14,336 bytes | full 128K |
phi-4 keeps a full cache on all 40 of its layers. Gemma 4 12B uses sliding-window attention on most layers, which only remembers a fixed, small window, so only its global layers grow with context. Qwen3.5-9B is a hybrid in which only 8 attention layers keep a KV cache. Mellum2.1 has 7 global layers and 21 sliding-window layers. For long documents or long chats on 12 GB, a model with a small per-token cache will get you much further than one with a smaller file.
Popular models that don’t fit
These need more memory than 12 GB under the fit rule. The last column shows how far short they fall.
| Model | Q4 file | Short by |
|---|---|---|
| gpt-oss-20b | 11.67 GB | 1.2 GB |
| Mistral Small 3.2 24B / Devstral Small | 14.33 GB | 3.9 GB |
| LFM2.5-24B-A2B | 14.53 GB | 4.1 GB |
| Gemma 4 26B A4B | 17.04 GB | 6.6 GB |
| Qwen3.8-27B | 17.44 GB | 7 GB |
| Qwen3.6-35B-A3B | 22.29 GB | 11.8 GB |
gpt-oss-20b is the near miss. It needs 13.2 GB, and any 16 GB card runs it with its full 128K context. You can push part of a model into system RAM, but that runs much slower than any number on this site, and we don’t estimate it.
Which 12 GB card to buy
The site’s speed estimates start from memory bandwidth, which is how fast the card can read its own memory. The table uses Gemma 4 12B as the test model. Prices are from Oct 19, 2026.
| Card | Price (Oct 19, 2026) | Bandwidth | Est. tok/s, Gemma 4 12B | $ per est. tok/s |
|---|---|---|---|---|
| RTX 3060 12GB | $301 used | 360 GB/s | 42 | $7.17 |
| Intel Arc B580 | $330 new | 456 GB/s | 38 | $8.68 |
| RTX 4070 SUPER | $612 used | 504 GB/s | 58 | $10.55 |
| RTX 5070 | $800 new | 672 GB/s | 76 | $10.53 |
The used RTX 3060 12GB gives the most estimated speed per dollar. The $301 price is the bottom of the typical range from 307 completed US eBay sales over 14 days (tracked by gpusold.com), and the median sale was $325. The RTX 4070 SUPER price is used too (median $639). The B580 and RTX 5070 prices are the lowest in-stock Newegg listings.
The B580 has more bandwidth than the 3060, yet it estimates slower on dense models, because Intel cards turn less of their bandwidth into speed in the published llama.cpp results our estimates are fitted to (0.59 of bandwidth, against 0.88 for NVIDIA with CUDA). On tiny models the order flips: on MiniCPM-V 4.7 1B the B580 estimates 204 tok/s to the 3060’s 175. Intel estimates are also rougher and can be off by 20 to 25% in either direction.
The fastest 12 GB card on our list is the RTX 3080 Ti, with 912 GB/s and an estimated 99 tok/s on Gemma 4 12B. Our price data has no listing for it.
You can see every model’s fit, context and estimated speed on one card with the My Rig page set to an RTX 3060 12GB.
When to step up from 12 GB
Twelve GB stops being enough when you want gpt-oss-20b, any of the 24B to 35B models above, or more than 9K of default context on a 14B dense model like phi-4.
The cheapest 16 GB card we price is the AMD Radeon RX 9070 at $570 new (Oct 19, 2026), with 640 GB/s. It runs gpt-oss-20b with the full 128K context at an estimated 154 tok/s.
The other route is two used RTX 3060 12GB cards, for 24 GB at about $602 (2 x $301). The model gets split by layers across the cards, which gives it more room. It doesn’t make it faster. In published llama-bench runs (llama.cpp’s speed test, posted on ahelpme.com), one, two and three RTX 3090s ran Qwen3.6-27B Q4_K_M at 40.1, 40.6 and 39.5 tok/s. So the speeds below are one card’s estimated speed:
| Model on two RTX 3060 12GB | Context (FP16 / 8-bit) | Est. tok/s |
|---|---|---|
| Gemma 4 26B A4B | full 256K | 104 |
| gpt-oss-20b | full 128K | 124 |
| Qwen3.6-35B-A3B | 33K / 63K | 100 |
| Devstral Small | 51K / 97K | 22 |
| Qwen3.8-27B | 82K / 155K | 17 |
The MoE models stay fast on this pair, while the dense Devstral Small and Qwen3.8-27B drop to an estimated 22 and 17 tok/s. Before you buy a second card, check your board and PSU: you need a motherboard with room for both cards and a power supply that can feed them.