FULL FINE-TUNING vs. LoRA:

Full Fine-Tuning:
├── Updates ALL model weights (billions of parameters)
├── Needs: 4-8 GPUs × 80 GB VRAM each
├── Cost: $10,000-100,000+ per training run
├── Time: Days to weeks
└── Verdict: Impossible for 99.9% of Pakistani developers

LoRA (Low-Rank Adaptation):
├── Freezes ALL original weights
├── Injects tiny trainable matrices ("adapters")
├── Updates ONLY the adapters (< 1% of parameters)
├── Needs: 1 GPU with 6-16 GB VRAM
├── Cost: PKR 0-500 per training run
├── Time: 30 minutes to 4 hours
└── Verdict: Your RTX 3060 can do this tonight

QLoRA (Quantized LoRA):
├── Same as LoRA BUT loads base model in 4-bit
├── Cuts VRAM usage by 50-75%
├── Needs: 1 GPU with 6-8 GB VRAM
├── Quality: 95-99% of full LoRA
└── Verdict: Your RTX 3060 (12 GB) or even RTX 3050 (8 GB)

THE LoRA DECOMPOSITION:

Original weight matrix W:    4096 × 4096 = 16,777,216 parameters
                              (frozen — don't touch)

LoRA adds two tiny matrices:
  B: 4096 × r    (r = rank, typically 4-32)
  A: r × 4096

For rank 8:
  B: 4096 × 8   = 32,768 parameters
  A: 8 × 4096   = 32,768 parameters
  Total adapter:   65,536 parameters (0.39% of original!)

The weight update: ΔW = B × A

                    ┌─────────────────┐
                    │  Original W      │ (frozen, 16M params)
                    │  4096 × 4096     │
                    └────────┬────────┘
                             │
                    ┌────────┴────────┐
                    │  + (B × A)      │ (trainable, 65K params)
                    │                  │
                    │  B: 4096×8       │
                    │  A: 8×4096       │
                    └─────────────────┘

YOU TRAIN 0.39% OF THE MODEL
THE OTHER 99.61% STAYS FROZEN

pip install transformers peft bitsandbytes datasets accelerate
# For QLoRA on Windows, you may need:
pip install bitsandbytes-windows

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# QLoRA: Load base model in 4-bit
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # Normalized Float 4
    bnb_4bit_compute_dtype="bfloat16",   # Computation in bf16
    bnb_4bit_use_double_quant=True       # Double quantization saves more VRAM
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

# Attach LoRA adapter
lora_config = LoraConfig(
    r=8,                              # Rank — start low, increase if needed
    lora_alpha=16,                    # Scaling factor (usually 2× rank)
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,                # Regularization
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 3,407,872 || all params: 8,030,261,248 || trainable%: 0.0424

from transformers import TrainingArguments, Trainer
from datasets import load_dataset

# Load your dataset (Alpaca format)
dataset = load_dataset("json", data_files="training_data.json")

training_args = TrainingArguments(
    output_dir="./lora-output",
    num_train_epochs=3,              # 3 epochs is usually enough
    per_device_train_batch_size=4,   # Adjust based on VRAM
    gradient_accumulation_steps=4,    # Effective batch = 16
    learning_rate=2e-4,              # LoRA uses higher LR than full FT
    warmup_steps=100,
    logging_steps=25,
    save_strategy="epoch",
    fp16=True,                       # Mixed precision
    report_to="none",                # Or "wandb" for W&B monitoring
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
)

trainer.train()

# Save just the adapter (tiny file, ~10-50 MB)
model.save_pretrained("./my-pakistan-adapter")

from peft import PeftModel

# Load base model + your adapter
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "./my-pakistan-adapter")

# Generate text with your fine-tuned model
inputs = tokenizer("DHA Phase 5 mein plot ka rate kya hai?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))

CHANNEL 1: CUSTOMER SERVICE BOTS (PKR 15,000-40,000/month)
├── Fine-tune on: Company FAQ + past chat logs + product catalog
├── Data needed: 500-1,000 Q&A pairs
├── Training time: 2-3 hours on RTX 3060
├── Result: Bot understands "delivery kab ayegi?" and "size chart dikhao"
├── Clients: Karachi retail, F&B, ecommerce sellers
└── Revenue: PKR 15-40K/month per client, 5 clients = PKR 75-200K/month

CHANNEL 2: LEGAL/REGULATORY DOCUMENT PROCESSING (PKR 30,000-80,000/project)
├── Fine-tune on: SECP filings, FBR forms, NADRA documents, contract templates
├── Data needed: 200-500 document examples
├── Training time: 3-4 hours
├── Result: Model extracts entities, summarizes clauses, flags risks
├── Clients: Law firms, corporate registrars, tax consultants
└── Revenue: PKR 30-80K per project, recurring monthly retainer

CHANNEL 3: HR SCREENING FOR LOCAL JOB PORTALS (PKR 20,000-50,000/month)
├── Fine-tune on: Pakistani CV formats, Rozee.pk job postings, salary norms
├── Data needed: 1,000+ CV-job pairs with match scores
├── Training time: 2-3 hours
├── Result: Model screens CVs understanding "NUST grad" vs "LUMS grad" context
├── Clients: Recruitment agencies, HR departments
└── Revenue: PKR 20-50K/month per client

MODEL + QUANTIZATION → VRAM NEEDED:

Llama 3 8B:
├── Full precision (FP16):     16 GB → RTX 4080 or better
├── 8-bit quantized:           10 GB → RTX 3060 12GB
├── 4-bit (QLoRA):              6 GB → RTX 3050 8GB ✓
└── Training with QLoRA rank 8: 8 GB → RTX 3060 12GB ✓

Llama 3 70B:
├── Full precision:            140 GB → 2× A100 80GB
├── 4-bit (QLoRA):             40 GB → A100 80GB or A6000 48GB
└── Training with QLoRA:       48 GB → Rented GPU only

Qwen 2.5 7B:
├── 4-bit (QLoRA):              5 GB → RTX 3050 8GB ✓
└── Training with QLoRA:        7 GB → RTX 3060 12GB ✓

PAKISTANI HARDWARE OPTIONS:
├── RTX 3060 12GB: PKR 65,000-85,000 (Hafeez Centre Lahore / Saddar Karachi)
├── RTX 4060 8GB: PKR 80,000-100,000 (good for inference, tight for training)
├── RTX 3090 24GB: PKR 150,000-180,000 (gold standard for local training)
├── Google Colab T4: Free (15 GB VRAM, limited hours)
├── Google Colab A100: PKR 2,800/month (Colab Pro+)
└── PaperSpace Gradient: PKR 2,200/month (A4000 16GB)