■ space · prose lane

THE TWO COMMANDS

# in one terminal: load a model
ollama run llama3.2 "hello"

# in another: see where it runs
ollama ps

NAME             ID           SIZE     PROCESSOR    UNTIL
llama3.2:latest  a80c4de17cd9 3.3 GB   100% GPU     4 minutes from now

Then read the server log, which names the hardware Ollama actually found at startup:

journalctl -u ollama --no-pager | grep -i "inference compute"

On a healthy NVIDIA box you want a line like:

inference compute id=GPU-xxxx library=CUDA compute=8.9 driver=12.4 name=NVIDIA GeForce RTX 4070

No line at all, or one that ends with a CPU-only fallback message, and you've found your problem. Detection happens once at startup, not per request — after any fix, sudo systemctl restart ollama before re-checking.

THE FIX TABLE

Work the causes in order — nine times out of ten the log line already told you which row you're in:

NVIDIA: THE DRIVER, NOT CUDA

The usual culprit is the driver, not CUDA. Ollama ships its own CUDA runtime libraries, so you don't need the CUDA toolkit — but the bundled runtime needs a driver new enough to talk to it. nvidia-smi working is not proof: it only proves a driver exists, not that it's recent enough.

nvidia-smi --query-gpu=driver_version --format=csv,noheader

# Debian/Ubuntu family
sudo apt install nvidia-driver-570
# Arch family
sudo pacman -S nvidia

# then re-detect:
sudo systemctl restart ollama

verify: journalctl -u ollama --no-pager | grep -i "inference compute" → library=CUDA

THE SPLIT: PARTIAL OFFLOAD IS MATH, NOT A BUG

Model weights plus the KV cache must fit in VRAM. Same model, bigger context — the cache outgrows the card and the rest spills to CPU:

Three ways to close the gap, cheapest first: a smaller quant (Q8 → Q4 halves the weights for a modest quality cost); a shorter context (num_ctx dominates the cache size — 32K on an 8 GB card means most layers stay on CPU); and fewer GPU layers — the num_gpu option caps offloads, and setting it below the layer count guarantees a split. If someone set it in a Modelfile or API call, unset it. One more trap: 100% GPU but still slow — the model may be swapping over system RAM; check ollama ps SIZE against your actual VRAM.

AMD ON LINUX: THREE CHECKABLE THINGS

1. ROCm in the build — the official install script bundles it. Confirm what the server detected: journalctl -u ollama --no-pager | grep -iE "rocm|inference compute".

2. Group membership — the runtime needs /dev/kfd + /dev/dri, i.e. the render/video groups:

sudo usermod -aG render,video $USER
# log out and back in, then:
sudo systemctl restart ollama

The missing group is the most common "Ollama not using GPU on Ubuntu" post on every forum — the driver was never the problem.

3. A supported GPU — or an override. Unsupported RDNA2 consumer cards (gfx1031, gfx1032) fail detection even with a working ROCm stack. The standard workaround is pretending to be a supported card: sudo systemctl edit ollama, then:

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

Then restart. Unsupported but widely used — if it misbehaves, remove it. On Windows, AMD support is narrower; check Ollama's supported-GPU list before assuming the install is broken.

verify: journalctl -u ollama --no-pager | grep -iE "rocm|inference compute" → your gfx target listed

"IT WORKED YESTERDAY"

Updates change one of three things, in order of likelihood:

1. A pinned backend library. OLLAMA_LLM_LIBRARY forces one runner (cuda_v11, rocm, even cpu) and silently beats autodetection. It's a debugging switch that persists in shell profiles and service files long after the reason is forgotten. Find it: systemctl show ollama --property=Environment | grep -i llm_library and env | grep OLLAMA — then remove it.
2. The driver fell behind the runtime. Ollama upgrades bundle a newer CUDA runtime; your driver doesn't move until you move it. Same fix as the driver section.
3. The service is a container and the flags are gone. A recreated Docker container without GPU flags is a CPU-only container — recreate with --gpus=all (or the AMD device flags).

end of the five causes · nothing fetched · everything local