Context length is the number of tokens a model can consider in a request or conversation under a given runtime configuration. A larger configured limit can increase KV-cache memory, prompt-processing time, and exposure to irrelevant or malicious text. The model’s advertised maximum does not mean every task needs it or that quality remains constant across the whole window.
Budget Tokens by Purpose
Break a request into system instructions, tool/schema definitions, conversation history, retrieved documents, user input, and reserved output. Token counts depend on the model tokenizer; character or word estimates are only rough.
Use a worksheet:
configured context limit
- fixed system/tool tokens
- reserved output tokens
- safety margin
= maximum dynamic input budget
Define truncation or summarization behavior before exceeding the budget. Silent left/right truncation can remove the user’s question, safety instruction, source, or earlier decision.
Understand the Resource Cost
Weights remain mostly fixed for a loaded model, while KV-cache demand grows with active context and sequences. Exact memory depends on architecture, layers, attention heads/dimensions, cache precision, runtime, and parallelism. Measure rather than applying a universal bytes-per-token number.
Long prompts also require more prefill computation. If an application retrieves 100 pages but only five contain useful evidence, increasing context may worsen latency and answer quality. Better retrieval, chunking, deduplication, and explicit citations can outperform brute-force context.
Test the Useful Context Envelope
Create evaluation tiers—for example short, typical, high, and maximum justified input. At each tier measure:
- factual retrieval from early, middle, and late positions;
- instruction priority and prompt-injection resistance;
- citation/attribution accuracy;
- TTFT, total latency, peak RAM/VRAM;
- concurrency and OOM behavior;
- truncation, refusal, and output completeness.
Use synthetic canary facts so no confidential data is needed. Keep model revision and sampling fixed.
Worked Example
A policy assistant sends an entire 200-page manual for every question. It is slow and sometimes cites the wrong section. The team indexes approved sections, retrieves a small set with source IDs, and reserves output space. Median context drops, while a fixed evaluation shows equal or better citation accuracy.
For exceptional questions, the system permits a larger bounded context and lower concurrency. Capacity planning uses both modes rather than advertising one maximum.
Failure Cases
- Setting maximum context because the model card lists it.
- Forgetting to reserve output tokens.
- Keeping unlimited conversation history.
- Truncating safety/system instructions.
- Sending duplicated or irrelevant retrieved chunks.
- Assuming “needle in a haystack” success equals real comprehension.
- Increasing context and concurrency without new memory tests.
🇵🇰 Pakistan Angle
Long bilingual documents can tokenize differently across English, Urdu script, and Roman Urdu. Count using the actual model tokenizer and evaluate each required language. Do not infer cost or capacity from English word counts.
When connectivity or power is constrained, smaller evidence packets reduce latency and compute. Preserve the source document locally only with authorization, encryption, access control, retention, and backup policy.
Hands-On Exercise
Take one document workflow. Create short/typical/high input fixtures with known answers at different positions. Measure token budget, TTFT, end-to-end latency, peak memory, accuracy, citations, and concurrency. Implement one retrieval or history-reduction change and rerun exactly.
Completion Rubric
- Complete: all token categories and output reserve are explicit, memory/latency/quality are measured by tier, and truncation is safe.
- Needs revision: context settings work but retrieval relevance, language tokenization, or concurrency effect is untested.
- Not complete: advertised maximum context is used as a quality or capacity guarantee.
Sources
- Ollama context length guidance
- Ollama FAQ: context and parallel request memory
- llama.cpp server context and KV-cache options
Key takeaway: Use the smallest evidence-complete context that passes the task, with output reserve and measured memory, latency, and retrieval quality.