GPU monitoring is useful when metrics are sampled over the same timeline as requests and interpreted against manufacturer/system limits. A single utilization or temperature number cannot diagnose performance or health. Never disable thermal protection or use a generic internet threshold as a substitute for hardware documentation.
Monitor a Minimal Set
Where supported, collect:
- GPU compute and memory utilization;
- used/total VRAM;
- temperature and fan behavior;
- board power and power limit;
- graphics and memory clocks, performance state;
- active processes;
- ECC/Xid or other hardware errors where available;
- application queue, latency, throughput, and errors.
Consumer GPUs may not expose every field. “Not supported” is not zero. Preserve sampling interval, units, device ID, driver version, and wall-clock timestamps.
For NVIDIA hardware, nvidia-smi provides read-only queries. Start with:
nvidia-smi
nvidia-smi -q -d MEMORY,UTILIZATION,TEMPERATURE,POWER,CLOCK
Consult the current documentation for query fields and looping/export options. Avoid configuration flags while learning; some actions require privileges and may change clocks, power, or device state.
Correlate, Don’t Guess
High compute utilization with stable clocks and acceptable latency may be healthy saturation. Low utilization may indicate small requests, CPU preprocessing, queue gaps, unsupported kernels, or measurement interval. High temperature plus falling clocks and worsening latency can suggest thermal throttling, but verify airflow, fan state, room conditions, dust, workload, and vendor thresholds.
VRAM near capacity is not itself an error if the runtime manages memory safely, but intermittent OOMs or degraded concurrency require headroom. Power readings must be interpreted with model workload and completion time.
Define Alerts and Stops
Set operational thresholds from manufacturer guidance, system design, baseline distribution, and service objectives. Use warning and critical states with duration—not one transient sample. Define who responds, safe shutdown procedure, and when professional hardware inspection is required.
Stop a test for burning smell, smoke, unusual electrical noise, fan failure, repeated hardware errors, temperature/power violations, unstable supply, or behavior outside documented limits. Do not open a powered PSU or improvise cooling around exposed electrical components.
Worked Example
Generation speed drops after 20 minutes. Monitoring shows rising temperature, decreasing clock, constant workload, and no memory growth. The operator stops within policy, checks blocked intake filters and room airflow, then retests after safe maintenance. The report calls this consistent with thermal throttling rather than proving component damage.
In another run, GPU utilization is low because requests arrive one at a time with long user pauses. Increasing batch size would harm interactive latency; low utilization is acceptable.
🇵🇰 Pakistan Angle
Record room temperature, dust-maintenance schedule, grid/UPS/inverter state, and whether air conditioning is part of the cost. Heat and power interruptions can change a sustained result. Do not stereotype all Pakistani environments; measure the actual room.
Design graceful shutdown for load-shedding and validate that the UPS/inverter can support the measured load under qualified electrical guidance. Monitoring software is not a substitute for safe wiring, earthing, and protection.
Hands-On Exercise
Run a safe 20-minute approved inference workload. Sample GPU and application metrics, mark request phases, and plot or tabulate the timeline. Identify one correlation, one alternative explanation, warning/stop rules, and a safe follow-up test. Do not alter power/thermal controls.
Completion Rubric
- Complete: telemetry is time-aligned, limitations and vendor guidance are respected, and alerts have owners and safe actions.
- Needs revision: metrics are captured but not connected to workload or sustained behavior.
- Not complete: protections are bypassed, unsafe stress is applied, or a single temperature is called a diagnosis.
Sources
- NVIDIA System Management Interface documentation
- NVIDIA DCGM diagnostics and health monitoring
- Ollama FAQ: model placement and concurrency
Key takeaway: Monitor GPU and application behavior together, respect device-specific limits, and stop safely when evidence crosses a documented boundary.