CPU vs. GPU Inference: Where the Bottleneck Really Is

Measure at least:

1. Cold load: reading model files and allocating memory.
2. Prompt processing/prefill: processing all input tokens; parallel compute matters heavily.
3. Decode: generating tokens sequentially; memory bandwidth and kernel efficiency often matter.
4. Post-processing: sampling, detokenization, validation, and application logic.
5. Queue/network: waiting before inference or moving data between services.

Time to first token includes more than generation speed. Tokens per second describes a generation interval but not end-to-end user latency or output quality.

A CPU has a smaller number of flexible cores and access to system RAM, which may provide large capacity at lower cost. Optimized CPU backends can run quantized models acceptably for light workloads. A GPU offers many parallel execution units and high-bandwidth device memory, but VRAM is limited and the runtime must support that GPU/backend.

Partial offload keeps some layers on the GPU and others in system memory. It can make a model fit, but transfers and slower CPU work can reduce performance. Integrated-memory systems behave differently from a desktop with separate RAM and VRAM; report the architecture instead of comparing labels alone.

Use the same model revision, quantization, prompt set, context, output limit, and sampling across runs. Warm up, then collect multiple samples. Record CPU utilization by core, RAM, GPU utilization, VRAM, power, temperature, disk activity, and phase timings.

Interpret patterns cautiously:

• low GPU use plus high one-core CPU use may indicate CPU-side work or unsupported kernels;
• high VRAM with low compute could indicate memory pressure or waiting;
• slow first run but faster later runs may reflect model loading or caches;
• falling speed with rising temperature may indicate thermal/power throttling;
• high throughput but poor p95 latency may indicate excessive batching or queues.

Do not infer causation from one utilization percentage. Correlate timestamps and change one variable.