A stress test asks whether the complete service remains correct, responsive, and safe during expected and exceptional demand. It is not an attempt to maximize heat or publish a dramatic concurrency number. Test only systems you own or are authorized to load, with monitoring, stop limits, and recovery prepared.
Define the Load Model
Use production evidence or conservative scenarios to define arrival rate, concurrency, prompt/output length distribution, model mix, session behavior, and burst duration. Include steady state, expected peak, short burst, soak, and recovery. Separate closed-loop virtual users from open-loop request arrival; they answer different questions.
Declare service objectives before testing:
- successful response rate and schema validity;
- TTFT and end-to-end p50/p95/p99;
- queue time and rejection behavior;
- throughput/goodput;
- maximum memory, temperature, and power within manufacturer/system policy;
- recovery after the load stops or one dependency fails.
Quality-check a sample of responses. A server returning fast corrupted or empty output is not healthy.
Prepare Safe Guardrails
Back up configuration, use synthetic prompts, isolate the test environment where possible, and notify affected people. Confirm logs contain no secrets or PII. Set stop conditions for temperature, memory errors, filesystem exhaustion, repeated timeouts, power instability, fan failure, or unexpected behavior.
Use vendor monitoring and supported diagnostics. Do not override thermal protection, overclock a production machine, or run destructive diagnostics on an unsupported consumer device. NVIDIA DCGM capabilities and supported hardware vary; follow the current documentation rather than copying data-center commands blindly.
Execute in Stages
- One-request functional baseline.
- Small load to validate generator and observability.
- Step load until the service objective first fails.
- Expected-peak soak for a justified duration.
- Short controlled burst and queue/rejection check.
- Recovery and data-integrity verification.
Change only one capacity setting between runs. Capture server and load-generator timestamps, CPU/RAM/GPU/VRAM, temperature, power, disk, network, queue depth, errors, model residency, and process restarts.
Worked Example
A local API meets p95 latency at four concurrent requests but deteriorates sharply at eight. Telemetry shows the runtime accepts too many active sequences, expands KV-cache demand, and begins heavy offload. The fix is a bounded concurrency limit plus queue/rejection policy—not a claim that the server supports eight users.
After the change, the team reruns the same workload and verifies that excess requests receive a controlled response while accepted requests remain within objectives. Capacity is stated as a measured workload envelope with date and configuration.
Failure Cases
- Sending requests as fast as possible with no realistic distribution.
- Running load against production without authorization or isolation.
- Measuring averages while p95/p99 and errors collapse.
- Ignoring response correctness, retries, and dropped streams.
- Continuing after thermal, power, or hardware error limits.
- Load generator becoming the bottleneck.
- Calling one 10-minute run a reliability or 500,000-user proof.
🇵🇰 Pakistan Angle
Test degraded network and planned power transitions safely if they are real requirements. Never trigger unsafe backup-power conditions. Record whether a UPS/inverter, generator, or grid supply was active and have a clean shutdown plan.
For a small local business, the honest target may be two staff members and a nightly batch—not internet-scale concurrency. Measure that first. Infrastructure purchases should follow observed demand and service objectives.
Hands-On Exercise
Write and run an authorized staged test using synthetic data. Include a workload model, five objectives, monitoring dashboard, stop conditions, baseline, step test, 30-minute soak, recovery check, raw results, and one retest after a bounded change. Keep load within hardware safety policy.
Completion Rubric
- Complete: load represents users, objectives and safety stops are predeclared, correctness and tails are measured, and recovery is verified.
- Needs revision: throughput is useful but workload realism, resource telemetry, or controlled rejection is missing.
- Not complete: the test is unauthorized, thermally unsafe, destructive, or used to imply unmeasured population scale.
Sources
Key takeaway: A real load test finds a safe, correct service envelope under a defined workload and proves recovery—not an impressive request count without context.