VRAM is the GPU’s working memory. It must hold the portion of model weights placed on the GPU plus KV cache, activations/workspace, runtime allocations, and display/other-process use. If the total exceeds usable VRAM, the runtime may reject the request, offload work, reduce capacity, or crash. Capacity and performance are connected but not identical.
Create a Memory Budget
Use this conceptual budget:
usable VRAM
- model weights on GPU
- KV cache for context × concurrent sequences
- runtime workspace and temporary tensors
- framework/driver reservations
- display and other processes
= safety headroom
Do not plan to use every reported byte. Driver/runtime fragmentation and workload spikes need headroom. The exact amount is runtime-, model-, precision-, context-, and hardware-specific, so measure startup logs and peak allocation.
Weight memory depends on parameter count and representation, but that rough arithmetic is only the start. KV cache generally grows with active tokens, layers, dimensions, precision, and concurrent sequences. Increasing context or parallel requests can make a previously fitting model fail even when weights are unchanged.
Distinguish Capacity From Speed
More VRAM lets you fit larger weights, longer context, more concurrent requests, or higher precision. It does not automatically provide more compute or bandwidth. Two cards with the same VRAM can have very different generation rates, supported data types, power use, and software compatibility.
System RAM is not a transparent substitute for VRAM on discrete-GPU systems. CPU/GPU offload may enable a model but can introduce slower memory access and transfers. Unified-memory systems need their own measured budget because the CPU, GPU, and operating system share capacity.
Measure the Real Envelope
Start with one small, known-good request. Increase one dimension at a time:
- model/quantization;
- context tokens;
- output limit;
- number of parallel sequences;
- batch size or runtime features.
Record idle, post-load, prompt peak, decode peak, and post-request memory. Repeat after several requests to detect leaks or fragmentation. Capture the runtime version and settings because memory behavior changes between releases.
Worked Example
A model loads with 1 GB free VRAM at 4K context and one request. The operator enables four parallel sequences and a much longer context, then receives out-of-memory errors. The weights did not change; the active KV-cache and runtime requirement did.
The fix is chosen by need: lower concurrency, shorter context, smaller/quantized model, KV-cache optimization supported by the runtime, partial offload, or a higher-memory device. Random driver reinstallations do not solve an honestly insufficient budget.
Failure Cases
- Using model-file size as the VRAM requirement.
- Forgetting the display and other GPU processes.
- Configuring maximum advertised context for every request.
- Multiplying concurrency without accounting for active tokens.
- Treating shared/unified memory as directly comparable to dedicated VRAM.
- Removing headroom and accepting intermittent OOMs.
- Buying based only on VRAM while ignoring backend support and condition.
🇵🇰 Pakistan Angle
Seller listings may omit the exact VRAM capacity or confuse similarly named GPU variants. Verify using a live system tool, model number, serial/warranty status, and a sustained test—not a screenshot alone. Record whether the card is refurbished, repaired, mined on, or imported without local warranty when the seller discloses it.
Higher-VRAM cards can require a larger quality PSU, connectors, case clearance, airflow, and backup-power capacity. Add those to the budget. Never use improvised power connectors or exceed manufacturer ratings to save PKR.
Hands-On Exercise
Build a VRAM envelope for one model. Record idle memory, loaded weights, 2K/4K/8K context peaks where supported, one/two/four concurrent requests, failures, temperature, and output quality. Leave a documented safety margin and identify the first limiting configuration.
Completion Rubric
- Complete: weights, cache, workspace, concurrency, other processes, and headroom are measured separately enough to explain the limit.
- Needs revision: peak VRAM is recorded but context/concurrency/runtime settings are missing.
- Not complete: the recommendation equates file size or product name with safe usable capacity.
Sources
- Ollama FAQ: context, parallelism, and memory
- Ollama context length guidance
- llama.cpp server memory and cache options
Key takeaway: VRAM capacity is a workload budget across weights, active context, concurrency, workspace, and headroom—not a single model-file comparison.