Tokens per second is useful only when its numerator, phase, settings, hardware, and quality are defined. Prompt-processing speed and generation speed are different. A model producing short wrong answers quickly is not better than a slower model that meets the task requirement.

// concept

Write a Reproducible Protocol

Record:

date, OS, runtime/build/commit, backend, driver, and library versions;
CPU, RAM, GPU/VRAM, storage, power mode, and other active processes;
exact model repository/revision, file digest, format, quantization, and license;
prompt template, context, input length, output limit, batch/parallel settings, and sampling;
cold/warm state, repetitions, random seed where supported, and quality rubric.

Do not mix results from different runtime versions or silently enable an optimization for only one candidate.

// concept

Measure the Right Phases

For interactive use, capture time to first token, inter-token latency, end-to-end latency, prompt tokens/second, generated tokens/second, and p50/p95 across repeated requests. For batch work, add total throughput and completion time. For serving, add request rate, concurrency, queue time, error rate, and goodput under a defined service-level objective.

Tokenizers differ, so raw tokens/second across different model families may not represent the same amount of text. Include characters or task items per second and a quality score when comparing families.

// concept

Build the Dataset

Use at least three workload classes: short prompt/short output, long prompt/short output, and realistic prompt/output. Randomize request order so warming or thermals do not favor one model. Remove confidential data and keep the fixture versioned.

Run a warm-up that is excluded from results. Repeat enough times to expose variance and report mean plus standard deviation or percentiles, not only the best run. Capture telemetry on the same timeline.

// worked_example

Worked Example

Model A generates 45 tokens/second and Model B 31. On a 50-item extraction set, A produces valid JSON 82% of the time while B produces 98%. With one retry, A’s latency and compute increase. The application values valid structured output, so B may provide higher effective goodput.

The report includes exact prompts, revisions, runtime settings, p50/p95, quality, memory, temperature, and failures. It does not publish “Model A is 45% better” from one speed ratio.

// concept

Use Tools Carefully

llama-bench can report average tokens/second and standard deviation for llama.cpp configurations. vLLM provides latency, throughput, and online serving benchmark commands. Use the command version shipped with the recorded release; flags change. Keep raw output and the exact command, but independently validate that the benchmark represents your application.

// failure_cases

Failure Cases

7 cases to diagnose

Reporting only the fastest run.
Comparing different prompt lengths or output limits.
Ignoring quality, invalid responses, or retries.
Comparing tokenizer-dependent speed without text/task normalization.
Benchmarking at unsafe temperatures or altered power settings.
Omitting runtime commit, quantization, or GPU placement.
Using synthetic maximum throughput as a user-latency claim.

// pakistan_angle

Pakistan Angle

Benchmark at the ambient and power conditions in which the machine will operate. Record any UPS/inverter state and do not create unsafe load tests during unstable supply. A screenshot from an air-conditioned international lab is not proof of sustained performance in your room.

If publishing buyer guidance, date every result and state the exact locally available hardware condition. Do not generalize one used card to all units of that model.

// hands_on

Hands-On Exercise

Benchmark two approved models or quantizations on 30 versioned prompts. Run five measured repetitions after warm-up. Produce raw results and a summary with TTFT, generation rate, end-to-end p50/p95, memory, temperature, error rate, task score, and effective successful tasks/minute.

// completion_rubric

Completion Rubric

3 grading bands

Complete
hardware/software/model/settings are reproducible, multiple phases and variance are reported, and quality changes the recommendation.
Needs revision
speed is measured consistently but quality, raw commands, or thermal context is incomplete.
Not complete
a best-case tokens/second screenshot is used as a model or purchase verdict.

// sources

Sources

3 official sources — check every claim yourself