An out-of-memory error means an allocation could not be satisfied in a particular memory domain at that moment. Diagnose whether the shortage is GPU VRAM, system RAM, container/cgroup limit, process address space, or disk caused by swapping—not every OOM is the same.

// concept

Preserve the Evidence

Capture the full error, timestamp, model/revision/quantization, runtime, context, input/output tokens, batch/concurrency, cache settings, GPU placement, active models, and memory before load/at failure. Record other GPU/CPU processes and whether the failure is immediate, size-dependent, concurrency-dependent, or appears after repeated requests.

Do not begin by clearing random caches or killing processes on a shared machine. Identify owners and follow operational policy.

// concept

Use an OOM Decision Tree

Does a tiny known-good model/request run? If no, investigate installation/device/environment.
Does the target model load? If no, reduce weight memory via supported quantization/offload or choose suitable hardware/model.
Does it fail with longer context/output? Reduce context, retrieval volume, output reserve, or supported cache precision.
Does it fail only concurrently? Limit active sequences, total tokens, batch size, or loaded models.
Does usage grow after requests? Check framework lifecycle, retained objects, sessions, cache eviction, fragmentation, or a leak.
Is system RAM/swap exhausted? Reduce offload, close authorized processes, add appropriate RAM, or choose a smaller model.

Retest the same failing fixture after each single change.

// concept

Choose the Least-Risky Fix

Possible fixes include smaller/quantized model, shorter context, better retrieval, lower concurrency, bounded queue, fewer resident models, reduced batch, supported KV-cache settings, partial offload, process restart as an incident containment, runtime update after review, or capacity increase.

“Empty cache” calls may release reusable allocator blocks but not live tensors or a true leak. They are not a capacity plan. Lowering precision or cache type can alter quality/numerics; run the evaluation again.

// worked_example

Worked Example

A server runs one request but OOMs after the fourth concurrent long conversation. Memory snapshots show weight memory stable and KV-cache rising with active tokens. The service adds per-request token limits, caps active sequences, bounds the queue, and tests overload response. No driver change is required.

Another process grows after every completed job because application code retains response objects. Reducing context delays failure but does not fix the leak. A heap/object investigation and lifecycle correction does.

// failure_cases

Failure Cases

7 cases to diagnose

Treating all OOMs as insufficient GPU VRAM.
Deleting caches without recording settings/evidence.
Increasing virtual memory until the service becomes unusably slow.
Lowering precision without quality retest.
Killing unknown shared processes.
Restarting automatically in an infinite crash loop.
Claiming a one-request pass proves production capacity.

// pakistan_angle

Pakistan Angle

Before purchasing hardware, reproduce the OOM with synthetic data and test configuration fixes. A capacity issue tied to rare long prompts may be cheaper to solve with limits and retrieval than a new GPU. If hardware is required, use the full rig and power procurement framework.

Power interruptions can leave jobs retried concurrently after restart, causing a thundering herd and OOM. Restore queues gradually, deduplicate/idempotently resume jobs, and verify model readiness before admitting traffic.

// hands_on

Hands-On Exercise

Create a safe OOM reproduction or simulated incident. Build a timeline and memory table, follow the decision tree, test at least three one-variable changes, and choose a fix. Add regression tests for long context, concurrency, repeated requests, overload, and restart recovery.

// completion_rubric

Completion Rubric

3 grading bands

Complete
memory domain and trigger are proven, fixes are isolated/retested, quality and overload remain safe, and a regression test exists.
Needs revision
the error stops but root cause, leak behavior, or safe capacity is uncertain.
Not complete
random process killing, cache deletion, or hardware purchase replaces diagnosis.

// sources

Sources

3 official sources — check every claim yourself