Start with the cheapest reversible intervention that fixes the measured error. Prompting changes instructions and examples at inference. Retrieval supplies current or private evidence. Fine-tuning changes model behavior from training examples, often through parameter-efficient adapters. None automatically adds truthful knowledge or removes the need for evaluation.
Diagnose the Error Type
Use a labeled failure set:
- unclear task/format: improve prompt, schema, examples, or deterministic validation;
- missing/changing facts: use retrieval or tools with source citations;
- repeated style/behavior mismatch: consider prompting first, then an adapter if consistent examples prove the need;
- insufficient base capability: select a stronger suitable model rather than training around fundamental failure;
- safety/policy failure: add layered controls and review; fine-tuning alone is not a safety boundary.
Do not fine-tune because it sounds advanced. It adds dataset governance, compute, experiment tracking, overfitting risk, artifact storage, license questions, and deployment complexity.
Establish Baselines
Freeze train/development/test splits before iteration. Remove duplicates and near-duplicates across splits. Document data source, consent/authorization, license, personal/sensitive fields, labeling guide, reviewer agreement, retention, and deletion. Use a held-out test that training and prompt authors cannot tune against repeatedly.
Run three baselines: simple instruction, improved prompt with examples, and retrieval/tool version if facts are the issue. Record task quality, critical errors, latency, context cost, and operating complexity.
Consider Parameter-Efficient Tuning
PEFT methods such as LoRA train a smaller set of adapter parameters while keeping base weights frozen, reducing training memory and artifact size. Quantized training approaches can reduce resource needs further, but hardware/software support and quality must be validated. “Fits on one GPU” is not a universal promise.
Pin the base model/revision, tokenizer, adapter configuration, libraries, seed, dataset version, and license. Track checkpoints and validation, then evaluate the combined base+adapter artifact. Never describe an adapter without its required base model and terms.
Worked Example
A ticket router needs one of eight exact labels. A clear prompt plus JSON schema raises valid outputs, but two niche categories remain confused. Retrieval is irrelevant because the problem is category boundaries. The team has 2,000 authorized, consistently labeled examples and trains a small adapter.
The held-out test improves those categories without worsening Urdu tickets or safety cases. Latency and memory remain acceptable. If the gain had been small or brittle, the prompt baseline would remain production choice.
Failure Cases
- Training on production conversations without consent or redaction.
- Using test examples during prompt/training iteration.
- Fine-tuning to memorize current facts.
- Believing more examples fix inconsistent labels.
- Ignoring base-model and dataset licenses.
- Publishing only average score while a critical slice worsens.
- Losing reproducibility for the final adapter.
🇵🇰 Pakistan Angle
Local-language data is valuable only when lawfully collected, representative of the intended workflow, and reviewed by fluent annotators. Roman Urdu spelling varies; write labeling rules and test robustness rather than forcing one “correct” spelling.
Cloud GPU training may expose data or create foreign-currency cost. Local training may be slower and power-sensitive. Compare both under approved data handling and dated PKR assumptions. Synthetic data must be validated and cannot replace real error coverage automatically.
Hands-On Exercise
Take 100 labeled, authorized examples and create frozen splits. Run a prompt baseline and, if facts matter, a retrieval baseline. Write a fine-tuning decision memo with error taxonomy, expected mechanism, data/license/privacy checks, compute estimate, acceptance thresholds, rollback, and “do not tune” outcome.
Completion Rubric
- Complete: the intervention matches the error, baselines and held-out test exist, data is governed, and fine-tuning is reproducible and reversible.
- Needs revision: training may help but splits, licensing, slice evaluation, or operating cost is incomplete.
- Not complete: fine-tuning is used to memorize facts, ingest unauthorized data, or bypass safety review.
Sources
- Hugging Face Transformers: parameter-efficient fine-tuning
- Hugging Face PEFT quantization guide
- Hugging Face dataset-card documentation
- NIST AI Risk Management Framework
Key takeaway: Prompt for instructions, retrieve changing evidence, and fine-tune only a proven behavioral gap with governed data and held-out evaluation.