Tokens per second is useful only when its numerator, phase, settings, hardware, and quality are defined. Prompt-processing speed and generation speed are different. A model producing short wrong answers quickly is not better than a slower model that meets the task requirement.
Write a Reproducible Protocol
Record:
- date, OS, runtime/build/commit, backend, driver, and library versions;
- CPU, RAM, GPU/VRAM, storage, power mode, and other active processes;
- exact model repository/revision, file digest, format, quantization, and license;
- prompt template, context, input length, output limit, batch/parallel settings, and sampling;
- cold/warm state, repetitions, random seed where supported, and quality rubric.
Do not mix results from different runtime versions or silently enable an optimization for only one candidate.
Measure the Right Phases
For interactive use, capture time to first token, inter-token latency, end-to-end latency, prompt tokens/second, generated tokens/second, and p50/p95 across repeated requests. For batch work, add total throughput and completion time. For serving, add request rate, concurrency, queue time, error rate, and goodput under a defined service-level objective.
Tokenizers differ, so raw tokens/second across different model families may not represent the same amount of text. Include characters or task items per second and a quality score when comparing families.
Build the Dataset
Use at least three workload classes: short prompt/short output, long prompt/short output, and realistic prompt/output. Randomize request order so warming or thermals do not favor one model. Remove confidential data and keep the fixture versioned.
Run a warm-up that is excluded from results. Repeat enough times to expose variance and report mean plus standard deviation or percentiles, not only the best run. Capture telemetry on the same timeline.
Worked Example
Model A generates 45 tokens/second and Model B 31. On a 50-item extraction set, A produces valid JSON 82% of the time while B produces 98%. With one retry, A’s latency and compute increase. The application values valid structured output, so B may provide higher effective goodput.
The report includes exact prompts, revisions, runtime settings, p50/p95, quality, memory, temperature, and failures. It does not publish “Model A is 45% better” from one speed ratio.
Use Tools Carefully
llama-bench can report average tokens/second and standard deviation for llama.cpp configurations. vLLM provides latency, throughput, and online serving benchmark commands. Use the command version shipped with the recorded release; flags change. Keep raw output and the exact command, but independently validate that the benchmark represents your application.
Failure Cases
- Reporting only the fastest run.
- Comparing different prompt lengths or output limits.
- Ignoring quality, invalid responses, or retries.
- Comparing tokenizer-dependent speed without text/task normalization.
- Benchmarking at unsafe temperatures or altered power settings.
- Omitting runtime commit, quantization, or GPU placement.
- Using synthetic maximum throughput as a user-latency claim.
🇵🇰 Pakistan Angle
Benchmark at the ambient and power conditions in which the machine will operate. Record any UPS/inverter state and do not create unsafe load tests during unstable supply. A screenshot from an air-conditioned international lab is not proof of sustained performance in your room.
If publishing buyer guidance, date every result and state the exact locally available hardware condition. Do not generalize one used card to all units of that model.
Hands-On Exercise
Benchmark two approved models or quantizations on 30 versioned prompts. Run five measured repetitions after warm-up. Produce raw results and a summary with TTFT, generation rate, end-to-end p50/p95, memory, temperature, error rate, task score, and effective successful tasks/minute.
Completion Rubric
- Complete: hardware/software/model/settings are reproducible, multiple phases and variance are reported, and quality changes the recommendation.
- Needs revision: speed is measured consistently but quality, raw commands, or thermal context is incomplete.
- Not complete: a best-case tokens/second screenshot is used as a model or purchase verdict.
Sources
Key takeaway: A defensible speed benchmark fixes the workload and reports latency, variance, quality, errors, and system conditions alongside tokens per second.