CPU and GPU inference differ in parallel compute, memory bandwidth, memory capacity, software support, and data-transfer cost. A GPU often accelerates dense model operations, but “GPU is faster” is not a complete diagnosis. Loading, prompt processing, token generation, CPU sampling, storage, network, context size, and thermal throttling can each dominate a run.
Split the Request Into Phases
Measure at least:
- Cold load: reading model files and allocating memory.
- Prompt processing/prefill: processing all input tokens; parallel compute matters heavily.
- Decode: generating tokens sequentially; memory bandwidth and kernel efficiency often matter.
- Post-processing: sampling, detokenization, validation, and application logic.
- Queue/network: waiting before inference or moving data between services.
Time to first token includes more than generation speed. Tokens per second describes a generation interval but not end-to-end user latency or output quality.
Understand the Hardware Paths
A CPU has a smaller number of flexible cores and access to system RAM, which may provide large capacity at lower cost. Optimized CPU backends can run quantized models acceptably for light workloads. A GPU offers many parallel execution units and high-bandwidth device memory, but VRAM is limited and the runtime must support that GPU/backend.
Partial offload keeps some layers on the GPU and others in system memory. It can make a model fit, but transfers and slower CPU work can reduce performance. Integrated-memory systems behave differently from a desktop with separate RAM and VRAM; report the architecture instead of comparing labels alone.
Find the Bottleneck With Evidence
Use the same model revision, quantization, prompt set, context, output limit, and sampling across runs. Warm up, then collect multiple samples. Record CPU utilization by core, RAM, GPU utilization, VRAM, power, temperature, disk activity, and phase timings.
Interpret patterns cautiously:
- low GPU use plus high one-core CPU use may indicate CPU-side work or unsupported kernels;
- high VRAM with low compute could indicate memory pressure or waiting;
- slow first run but faster later runs may reflect model loading or caches;
- falling speed with rising temperature may indicate thermal/power throttling;
- high throughput but poor p95 latency may indicate excessive batching or queues.
Do not infer causation from one utilization percentage. Correlate timestamps and change one variable.
Worked Example
A 4-bit model runs at acceptable decode speed on a GPU, but users wait twelve seconds for the first token. Tracing shows the service unloads the model after each request and reloads it from a slow disk. Buying a faster GPU would not fix the lifecycle problem. Keeping the model resident within safe memory and access limits improves latency.
Another system uses CPU-only inference overnight for batch classification because latency is unimportant and RAM capacity is sufficient. The same hardware is unsuitable for an interactive multi-user chat. Suitability belongs to the workload.
🇵🇰 Pakistan Angle
A used GPU may appear economical, but include the required power supply, motherboard lanes, case airflow, UPS/inverter capacity, cooling, warranty, and electricity. A CPU-only pilot on existing hardware may be the safest way to validate demand before importing or buying a card.
Test under realistic room temperature and power conditions without disabling thermal protections. Do not run questionable adapters or overloaded extension boards. Electrical and fire safety outrank benchmark speed; use qualified help for power design.
Hands-On Exercise
Benchmark one approved model in CPU-only and supported GPU/offload modes. Use ten fixed prompts and five repetitions after warm-up. Record phase timings, p50/p95 latency, tokens/second, CPU/RAM/GPU/VRAM, temperature, power where safely available, quality score, and errors. Identify the bottleneck and propose one bounded change.
Completion Rubric
- Complete: the comparison controls model/settings, separates phases, includes quality and percentiles, and links the recommendation to the workload.
- Needs revision: a speed number exists but utilization, warm-up, thermals, or end-to-end latency is missing.
- Not complete: the answer assumes a GPU upgrade without measurement or ignores electrical safety.
Sources
- llama.cpp supported CPU, GPU, and hybrid backends
- Ollama FAQ: checking CPU/GPU model placement
- NVIDIA System Management Interface documentation
Key takeaway: Diagnose the slow request phase with controlled measurements; the most visible component is not always the bottleneck.