silicon-layer
0/24 complete

Module 3: GPU Benchmarking · 20 min

Benchmarking Tokens per Second Across Models

// sabak

Turn this lesson into one checked practice output

By the end, you should be able to explain the core idea behind “Benchmarking Tokens per Second Across Models” in your own words, apply it to one small real or sample task, and identify what still needs human review.

  1. 1

    Learn

    Read the 20-minute lesson without copying an output blindly.

  2. 2

    Try

    Use a small, non-sensitive example that you can inspect line by line.

  3. 3

    Review

    Check facts, fit, and risk; save one improvement note for next time.

Tokens per second is useful only when its numerator, phase, settings, hardware, and quality are defined. Prompt-processing speed and generation speed are different. A model producing short wrong answers quickly is not better than a slower model that meets the task requirement.

Write a Reproducible Protocol

Record:

  • date, OS, runtime/build/commit, backend, driver, and library versions;
  • CPU, RAM, GPU/VRAM, storage, power mode, and other active processes;
  • exact model repository/revision, file digest, format, quantization, and license;
  • prompt template, context, input length, output limit, batch/parallel settings, and sampling;
  • cold/warm state, repetitions, random seed where supported, and quality rubric.

Do not mix results from different runtime versions or silently enable an optimization for only one candidate.

Measure the Right Phases

For interactive use, capture time to first token, inter-token latency, end-to-end latency, prompt tokens/second, generated tokens/second, and p50/p95 across repeated requests. For batch work, add total throughput and completion time. For serving, add request rate, concurrency, queue time, error rate, and goodput under a defined service-level objective.

Tokenizers differ, so raw tokens/second across different model families may not represent the same amount of text. Include characters or task items per second and a quality score when comparing families.

Build the Dataset

Use at least three workload classes: short prompt/short output, long prompt/short output, and realistic prompt/output. Randomize request order so warming or thermals do not favor one model. Remove confidential data and keep the fixture versioned.

Run a warm-up that is excluded from results. Repeat enough times to expose variance and report mean plus standard deviation or percentiles, not only the best run. Capture telemetry on the same timeline.

Worked Example

Model A generates 45 tokens/second and Model B 31. On a 50-item extraction set, A produces valid JSON 82% of the time while B produces 98%. With one retry, A’s latency and compute increase. The application values valid structured output, so B may provide higher effective goodput.

The report includes exact prompts, revisions, runtime settings, p50/p95, quality, memory, temperature, and failures. It does not publish “Model A is 45% better” from one speed ratio.

Use Tools Carefully

llama-bench can report average tokens/second and standard deviation for llama.cpp configurations. vLLM provides latency, throughput, and online serving benchmark commands. Use the command version shipped with the recorded release; flags change. Keep raw output and the exact command, but independently validate that the benchmark represents your application.

Failure Cases

  • Reporting only the fastest run.
  • Comparing different prompt lengths or output limits.
  • Ignoring quality, invalid responses, or retries.
  • Comparing tokenizer-dependent speed without text/task normalization.
  • Benchmarking at unsafe temperatures or altered power settings.
  • Omitting runtime commit, quantization, or GPU placement.
  • Using synthetic maximum throughput as a user-latency claim.

🇵🇰 Pakistan Angle

Benchmark at the ambient and power conditions in which the machine will operate. Record any UPS/inverter state and do not create unsafe load tests during unstable supply. A screenshot from an air-conditioned international lab is not proof of sustained performance in your room.

If publishing buyer guidance, date every result and state the exact locally available hardware condition. Do not generalize one used card to all units of that model.

Hands-On Exercise

Benchmark two approved models or quantizations on 30 versioned prompts. Run five measured repetitions after warm-up. Produce raw results and a summary with TTFT, generation rate, end-to-end p50/p95, memory, temperature, error rate, task score, and effective successful tasks/minute.

Completion Rubric

  • Complete: hardware/software/model/settings are reproducible, multiple phases and variance are reported, and quality changes the recommendation.
  • Needs revision: speed is measured consistently but quality, raw commands, or thermal context is incomplete.
  • Not complete: a best-case tokens/second screenshot is used as a model or purchase verdict.

Sources

Key takeaway: A defensible speed benchmark fixes the workload and reports latency, variance, quality, errors, and system conditions alongside tokens per second.

Self-check

Before you mark Lesson 3.1 complete

  • Can I explain “Benchmarking Tokens per Second Across Models” without reading the lesson back word for word?
  • Did I complete the lesson’s practice step on a real or clearly labelled sample task?
  • Did I check the result for invented facts, private data, unsafe actions, and mismatch with the brief?