CPU/GPU offloading places some model work or weights in GPU memory and the remainder in system memory/CPU. It can make a model run when it does not fully fit in VRAM, but transfers and slower CPU execution can reduce speed. Offloading is a capacity technique, not free extra VRAM.
Establish the Baseline
Record the exact model/revision, quantization, runtime build, context, batch/concurrency, hardware, and a fixed prompt set. Measure CPU-only and maximum safe GPU-resident configurations where possible. Capture TTFT, prompt/generation speed, p50/p95, output quality, RAM/VRAM, CPU/GPU utilization, power, and temperature.
In llama.cpp, GPU-layer controls and device options determine placement; names and behavior can change, so use the current server/CLI help for the pinned build. Ollama can report whether a model is on CPU, GPU, or split. Do not copy a magic layer count from different hardware.
Sweep Placement Systematically
Start from a known stable configuration and increase GPU-resident layers or use the runtime’s supported automatic fit. At each point leave headroom for context and concurrency. Test short and long inputs because KV-cache demand can change the fit.
Watch for:
- model load failure or OOM;
- system RAM pressure and swapping/pageouts;
- CPU saturation and per-core behavior;
- PCIe transfer/utilization where available;
- declining clocks or rising temperature;
- worse tail latency despite faster average decode;
- reduced concurrency from smaller headroom.
The fastest single request may not be the best server setting. A slightly lower offload can preserve room for a second sequence and produce higher useful throughput.
Worked Example
A quantized model does not fit fully in a 12 GB card at the required context. CPU-only generation is too slow. A layer sweep finds that heavy GPU offload improves decode but leaves almost no VRAM and fails on long requests. A moderate split is slower per request but passes the complete test set with two concurrent users.
The service adopts the moderate setting and caps tokens/concurrency. The report states exact hardware and workload instead of claiming that this model “runs on 12 GB.”
Understand System RAM
System RAM must hold offloaded weights, runtime data, OS, and application processes. If the OS begins swapping to disk, latency can become unstable and storage wear may increase. Memory mapping can affect load behavior but is runtime/OS-specific. Never use an oversized page file as proof that the system has adequate RAM.
Integrated/unified-memory devices have different data paths. Use their runtime’s documentation and measured telemetry rather than applying discrete-GPU assumptions.
Failure Cases
- Using a layer count from another model or runtime version.
- Filling VRAM before testing context/concurrency.
- Ignoring system RAM and swap.
- Comparing offload runs with different prompts/settings.
- Assuming automatic fit optimizes the business objective.
- Calling “loads successfully” a usable latency result.
- Overclocking or bypassing protections to recover speed.
🇵🇰 Pakistan Angle
Offloading can extend existing hardware and avoid an immediate purchase, but calculate staff waiting time and electricity as well as avoided GPU cost. A slow setup may still be excellent for overnight batch work and poor for a client-facing chat.
Test sustained behavior under the actual room and safe power setup. Higher CPU involvement adds system heat and power, so review whole-system cooling rather than GPU temperature alone.
Hands-On Exercise
Run a five-point offload sweep using one model and 20 prompts at two context tiers. Record placement, TTFT, speed, p95, quality, RAM/VRAM, swap, utilization, thermals, and two-request behavior. Choose one configuration with headroom and document its limit.
Completion Rubric
- Complete: the sweep is controlled, whole-system memory and tail latency are measured, and the selection fits context/concurrency needs.
- Needs revision: the model fits and speed improves but headroom, swap, or sustained behavior is unclear.
- Not complete: offloading is treated as free capacity or justified by load success alone.
Sources
- llama.cpp HTTP server device and GPU-layer options
- llama.cpp CPU/GPU hybrid inference
- Ollama FAQ: processor placement
Key takeaway: Offload only as much as produces the best complete workload result with RAM, VRAM, context, concurrency, and thermal headroom.