What GPU do you need to run Thinking Machines Inkling 2 Small?
No single GPU. Thinking Machines Inkling 2 Small needs about 355.6 GB at UD-Q4_K_XL, more than any GPU, system or build we price.
Early support Runs on llama.cpp pull request build and mlx-lm pull request. Stock llama.cpp, Ollama and LM Studio can't load Inkling 2 Small yet. The GGUFs use a new inkling2 architecture. Unsloth's guide (unsloth.ai/docs/models/inkling-2) builds llama.cpp from pull request #32187 ("model : add Inkling 2 (stacked on Inkling 1)", opened Nov 13, 2026, a draft and still open), and builds that support the original Inkling won't load it. On a Mac, mlx-community/Inkling-2-Small-4bit (uploaded Nov 13, 2026, 380,788,964,744 bytes) needs mlx-lm from unmerged pull request #2061 ("Add Inkling 2 model"). Its card says the routed experts keep NVFP4 (layers 0 to 39) or MXFP8 (layers 40 to 47) and the other weights are 4-bit, and it does not include the drafter. The fits and speeds on this page are math on the UD-Q4_K_XL file, not a test. (source, checked Nov 13, 2026.)
593.72B parameters, 17B active (mixture-of-experts) (Hugging Face, accessed 2026-11-13). UD-Q4_K_XL GGUF: 354.15 GB (source). No context estimate: Inkling 2 Small's inkling2 architecture is only in an open llama.cpp pull request, so there is no verified llama.cpp KV formula for it yet. Its config.json marks the 40 sliding-window (1,024-token) layers in local_layer_ids, which the generic config reading here does not handle (details).
Thinking Machines' second-generation Small model: a mixture-of-experts model with 594B total and 17B active parameters (the card's figures), text and image in, text out, trained in NVFP4. 593.72B is the total in unsloth's GGUF metadata (593,722,994,688); Hugging Face's own counter reads lower because NVFP4 packs two weights per byte. There is no plain Q4_K_M, so the size is unsloth's UD-Q4_K_XL (9 files, 354,150,089,945 bytes); every size is in the table below. Fit leaves out the 0.66 GB vision projector, which is extra if you send images. The checkpoint ships an MTP drafter for speculative decoding; the speed estimates on this page do not assume it. Release date is the day the weights went public.
What we checked
Checked Nov 13, 2026. Official is the maker or the weight files. A vendor claim is that project's own documentation. A community claim is someone else's upload. Our check is a file or search we read. Unknown means we have no source, so it is not estimated. None of these rows is a speed we measured.
| Kind | What | Detail | Source |
|---|---|---|---|
| Official | Model card | Thinking Machines' card: "594B total, 17B active", a context of "up to 1M tokens" (config model_max_length 1,048,576), text and image input with text output, and "pretrained, midtrained and post-trained in NVFP4". config.json (model_type inkling2): 48 layers, hidden size 4,608, 48 attention heads, 8 KV heads, head size 128; 40 local layers with a 1,024-token sliding window and 8 global layers; 384 routed experts, 8 per token, no shared expert; a vision tower; and an MTP drafter config with 5 layers. This page's parameter count is the GGUF metadata total of 593,722,994,688 in unsloth's GGUF. Hugging Face's safetensors counter shows 383.07B because NVFP4 packs two weights per byte; that is not the parameter count. | Model card, checked Nov 13, 2026 |
| Official | Release and license | Thinking Machines' announcement is dated Nov 5, 2026 and carries the line "Nov 12, 2026 Update: Inkling 2 Small's weights are now available for download on HuggingFace". The release date on this page is that Nov 12 update. License: Apache 2.0, per the repo's license tag and the announcement. This is a summary of the license, not legal advice. | Thinking Machines announcement, checked Nov 13, 2026 |
| Official | Weight files | 25 NVFP4 safetensors files sum to 393,480,084,520 bytes (393.48 GB) at commit aac974122acd33ddb0d3e1f54433a9c5dd2249ce, including mtp.safetensors (3,423,429,800 bytes), the MTP drafter. Thinking Machines also publishes an MXFP8 checkpoint, thinkingmachines/Inkling-2-Small-MXFP8; we did not read its size. These are downloads, not the memory a runtime needs. | Hugging Face API, checked Nov 13, 2026 |
| Vendor claim | Server hardware (Thinking Machines' figures) | Thinking Machines' card says the NVFP4 checkpoint "requires a GPU cluster with at least 500GB of aggregated VRAM", for example 2x B300 or 4x H200, and that the MXFP8 checkpoint needs at least 800GB. For local inference it lists SGLang, vLLM, TokenSpeed and Unsloth. Those are Thinking Machines' statements about the safetensors checkpoints. The fits on this page use the UD-Q4_K_XL GGUF. | Model card, checked Nov 13, 2026 |
| Vendor claim | Benchmark scores (Thinking Machines') | The card's eval results include SWE-bench Multilingual 81.2, GPQA Diamond 93 and MMMU Pro 81.6. These are Thinking Machines' own reported scores, not checked by us. | Model card, checked Nov 13, 2026 |
| Community claim | GGUF files (unsloth) | unsloth/Inkling-2-Small-GGUF (commit caea5b5c473a199102fa9f9f32363e70faa49059). UD-Q4_K_XL is 9 files totaling 354,150,089,945 bytes, and the fits on this page use it. Unsloth's card says its BF16 GGUF keeps the original NVFP4 experts bit-exact (layers 0 to 39). The card's KLD table is Unsloth's measurement, and this page does not list it. mmproj-F16.gguf (658,223,104 bytes) is the vision projector. Every size is in the table below. | unsloth GGUF card, checked Nov 13, 2026 |
| Our check | llama.cpp | Pull request #32187, "model : add Inkling 2 (stacked on Inkling 1)", was opened on 2026-11-13 as a draft and is still open. Unsloth's guide builds llama.cpp from it and says builds that support the original Inkling won't load Inkling 2. Stock llama.cpp, Ollama and LM Studio can't load it yet. | llama.cpp PR #32187, checked Nov 13, 2026 |
| Community claim | MLX (unmerged mlx-lm pull request) | mlx-community/Inkling-2-Small-4bit was uploaded on 2026-11-13 (380,788,964,744 bytes). It needs mlx-lm from pull request #2061, "Add Inkling 2 model", which is open and not merged, so MLX support is unofficial for now. The card says the routed experts keep NVFP4 (layers 0 to 39) or MXFP8 (layers 40 to 47) and the other weights are 4-bit. It does not include the drafter. | mlx-lm PR #2061, checked Nov 13, 2026 |
GGUF files and sizes
Exact sizes from Hugging Face, accessed 2026-11-13. The fit and context tables on this page use the UD-Q4_K_XL file (354.15 GB); a smaller file needs less memory by the difference. VRAM needed is the file + the same headroom.
| File | Published by | Bytes | Size | VRAM needed | Source |
|---|---|---|---|---|---|
| UD-IQ1_S | unsloth | 171,728,965,465 | 171.73 GB | ~173.2 GB | 5 parts |
| UD-IQ1_M | unsloth | 178,544,971,609 | 178.54 GB | ~180 GB | 5 parts |
| UD-IQ2_XXS | unsloth | 190,584,459,097 | 190.58 GB | ~192.1 GB | 5 parts |
| UD-Q2_K_XL | unsloth | 204,969,271,289 | 204.97 GB | ~206.5 GB | 6 parts |
| UD-IQ3_XXS | unsloth | 225,986,223,385 | 225.99 GB | ~227.5 GB | 6 parts |
| UD-Q3_K_XL | unsloth | 285,128,675,513 | 285.13 GB | ~286.6 GB | 7 parts |
| UD-Q4_K_XL | unsloth | 354,150,089,945 | 354.15 GB | ~355.6 GB | UD-Q4_K_XL/ (9 parts) |
| UD-Q8_K_XL | unsloth | 386,510,193,913 | 386.51 GB | ~388 GB | 9 parts |
| BF16 | unsloth | 484,382,454,681 | 484.38 GB | ~485.9 GB | 11 parts |
| mmproj-F16 Vision projector, extra | unsloth | 658,223,104 | 0.66 GB | — | mmproj-F16.gguf |
Other places to get it
Checked Nov 14, 2026. We list only copies we opened and checked. Check file hashes for copies from other uploaders.
- ModelScope: Other uploaderSame account name as on Hugging Face; we found no link from the lab to it
- Ollama: Cloud only (runs on Ollama's servers)
ollama run inkling-2-small:cloud
Weights-only fits, context unknown. We have no KV cache data for Thinking Machines Inkling 2 Small, so every fit on this page only checks that the UD-Q4_K_XL file and headroom fit. How much context fits beside the weights is unknown, so none of these is a proven fit for any context length.
No single GPU in our database fits it. Use multiple GPUs (see calculator).
Every device that runs it
Graphics cards, Macs and mini PCs from the wider hardware list behind My rig, smallest usable memory first. Not sure about yours? Check your exact rig.
Show all 1 devices that run it
Speed calculator
estimateEstimate = bandwidth ÷ GB read per token (the model's listed GGUF file: Q4_K_M for most models, otherwise the quant named in the list; scaled by active/total parameters for MoE), scaled by a per-platform efficiency and fixed per-token costs calibrated to 45 measured llama.cpp benchmarks (2025–26). Fit = file size + 1.5 GB headroom ≤ total VRAM (on Macs, the part of unified memory macOS lets the GPU use by default), and room for at least 4K tokens of context with an 8-bit KV cache. For a model with no KV cache data, only the file and headroom are checked, so a fit says nothing about context. Multi-GPU: VRAM adds, speed stays one card's (a layer split runs the cards in turn); max context uses the total. Speed is for short context. Max context = what is left after the file and a 1 GB runtime reserve ÷ KV cache bytes per token, default FP16 and 8-bit (q8_0) cache. How speed works · how context works