TL;DR — the answer first

Yes — vLLM supports GGUF, on GPUs only. The official quantization matrix (updated September 14, 2026) lists GGUF among 12 supported formats, green on every NVIDIA row from Volta (SM 7.0) to Hopper (SM 9.0) plus AMD GPU — and red on x86 CPU, Arm CPU and Intel GPU. On a CPU-only box the file belongs to llama.cpp or Ollama, which is exactly where our bench ran it: a 2B GGUF model answered 98% of warm runs in 0.8 s on 8 cores.

12quant formats on the official list — GGUF among them

6 of 9hardware columns green: Volta → Hopper + AMD GPU

0CPU rows — x86, Arm and Intel GPU all closed

01the matrix, verbatim

The whole contract, copied from the official quantization docs — six green columns, three red.

vLLM hardware support for GGUF — docs.vllm.ai, quantization matrix, 2026-09-14
hardware	GGUF in vLLM	note
Volta (SM 7.0)	✅	the oldest green row
Turing (SM 7.5)	✅
Ampere (SM 8.0/8.6)	✅	RTX 30xx, A100 territory
Ada (SM 8.9)	✅	RTX 40xx
Hopper (SM 9.0)	✅	H100
AMD GPU	✅	the only non-NVIDIA green
Intel GPU	❌
x86 CPU	❌	the row most searchers are really asking about
Arm CPU	❌

02the GPU-border rule

Supports the format and runs on your hardware are two different yeses.

vLLM grows no second runtime to digest GGUF. GGUF is a lane on the quantization list — weights in, same serving stack above it — and the lane is wired to GPU kernels only. Two consequences follow: the tokenizer is never optional (vLLM still reads the HF tokenizer files — pass them explicitly, as the FAQ hub's standing line goes), and loading is a lane choice, not a promise: a repo:quant_type tag or a single-file GGUF points vLLM at the lane, the matrix decides whether it opens.

fig 1 — the file is fine; the road decides. the CPU lane has no checkpoint to clear.

GGUF obeys the GPU-border rule: a passport vLLM only stamps at GPU checkpoints — and no CPU lane exists to queue at.

03why the CPU lane is closed

GGUF was born for CPU inference — in llama.cpp. The intuition transfers badly.

symptom → cause → fix
symptom	cause	fix
vLLM + a `.gguf` on a CPU-only box	no CPU kernel path — matrix rows are red by design	run the file with llama.cpp or Ollama, or serve a non-GGUF checkpoint through vLLM
GGUF loads, tokenizer errors at startup	HF tokenizer files never passed — GGUF bundles weights, not the HF set	point `tokenizer` at the model's HF repo
served fine on GPU, CPU fallback expected	the lane is GPU-bound; there is no offload story	keep vLLM on the GPU, give CPU boxes to the detour below

GGUF on CPU in vLLM is supported in the same sense a boat is supported by a highway: the file travels, the road disagrees.

04the CPU detour, with receipts

The detour is not a consolation prize — it is where GGUF actually lives on CPU.

run it yourself — verbatim from the 2026-10-24 bench

$ ollama run d3:2b "A customer says their card was charged twice." \
    --ask "Is this a billing issue? answer yes or no"
yes  98%

runtimeOllama is GGUF-native and moved again v0.40.11 · 2026-10-24 — llama.cpp is the format's home turf

receipt2B GGUF decision model: 98% of warm runs in 0.8 s, 8-core CPU box, no GPU line item — from the CPU bench ledger

keep vLLM forhigh-throughput GPU serving — the thing the green columns are actually for

the FAQ hubcan vLLM run GGUF — ten family questions, answered first, holds the plugin lane and the repo-quant syntax

picking a quantGGUF quantization levels — what Q4_K_M vs Q8_0 costs in size and fidelity

no GPU at all, Windows?vLLM on Windows runs through WSL2 — and the GGUF matrix above decides the CPU question either way

05the short answers

The three questions the search board actually holds, answered in one line each.

can vLLM use GGUF on CPU?no — x86 and Arm rows are ❌ in the official matrix; a hardware contract, not a configuration miss

can vLLM load a local .gguf?yes, on the GPU lane — local path or repo:quant_type tag, tokenizer passed alongside

why the tokenizer demand?vLLM's text stack reads HF tokenizer files; GGUF bundles weights its own way — pass the repo, never assume

■ honest stamp — the matrix is quoted, not benched: vLLM quantization docs, updated 2026-09-14, are the source for every green and red cell, and no vLLM-GGUF run was attempted here. the CPU receipt is our own bench ledger (2026-10-24: QEMU-KVM VM, Ubuntu 24.04, ollama 0.40.10, CPU-only, verbatim runs). ollama v0.40.11 per the GitHub release feed, 2026-10-24. the strike line that fed this page: "can vllm run gguf" — 18 impressions, position 6.8, zero clicks over 28 days in Google Search Console. the matrix is the contract until vLLM edits it; the day a CPU row turns green, this page ages on purpose.

— the matrix is the contract · Mr Say Nothing