Deploy a local inference service for one authorized, bounded project and produce evidence that another operator can run, secure, measure, recover, and retire it. The capstone is complete when it passes quality, security, reliability, and cost acceptance—not when a model responds once.
1. Write the Service Brief
Define user/task, approved data, excluded uses, model output contract, languages, quality rubric, critical failure classes, latency/concurrency objectives, hours of operation, owner, and stop conditions. Choose local only after comparing cloud/hybrid alternatives.
Create at least 75 synthetic or authorized redacted evaluation cases with frozen acceptance thresholds. Include malformed input, prompt injection, long input, bilingual cases where required, and policy refusal.
2. Select and Pin the Stack
Document host hardware, OS, driver, runtime/build, model repository/revision/digest, quantization, chat template, context, sampling, license/terms, and source. Verify model provenance and scan/review dependencies under organizational policy.
Keep an artifact manifest and exact configuration without secrets. Model aliases in the application map to pinned internal revisions. Define update review and rollback.
3. Build the Protected Service
Run the inference runtime on loopback or isolated service network. Put an application gateway in front with identity, authorization, TLS/private transport, model allowlist, body/token/output limits, bounded concurrency/queue, timeouts, cancellation, backoff, schema validation, and safe error messages.
Do not execute model-generated code/commands or expose arbitrary tools. Treat output as untrusted. Keep secrets outside source control and browser bundles.
4. Add Observability
Measure request IDs, model revision, accept/reject, queue time, TTFT, total latency, token counts, schema/quality samples, errors, CPU/RAM/GPU/VRAM, temperature/power, process/model residency, and restart events. Avoid raw prompt logging by default. Define warning/critical rules, owner, and runbook.
Health proves process status; readiness proves the pinned model can serve. Test that traffic waits during startup and stops during unsafe state.
5. Validate Performance and Recovery
Run functional baseline, controlled step load, expected-peak soak, overload/rejection, and recovery using realistic request distributions. Report p50/p95/p99, goodput, errors, quality, memory/thermal headroom, and exact workload.
Test safe shutdown, power-loss simulation only through approved non-destructive methods, process crash, full disk/log growth protection, model update rollback, queued-job idempotency, and configuration restore. Never bypass device protections.
6. Review Security and Privacy
Threat-model users, devices, network paths, model artifacts, prompt injection, output handling, logs, backups, and admin access. Test unauthorized device/user, revoked identity, oversized/malformed request, rate limit, cross-tenant cache isolation if applicable, and secret/log redaction.
Record data retention, deletion, backups, and incident response. Obtain qualified review for regulated or high-impact uses.
7. Prove Cost and Handoff
Build dated local/cloud/hybrid 24-month scenarios using actual measured energy, utilization, quality, demand, and staff assumptions. Create setup, start/stop, update, rollback, backup/restore, incident, and retirement runbooks. A second person should deploy or restore from the documentation.
🇵🇰 Pakistan Angle
Include dated PKR hardware, tariff, exchange, warranty, cooling, UPS/inverter, internet, and replacement-lead assumptions. Verify Urdu/Roman Urdu only if required and with fluent reviewers. Test actual room and safe power conditions without generalizing them to Pakistan.
Do not market the capstone as guaranteed savings, privacy, uptime, or income. State the exact tested device count, request distribution, duration, date, and limitations.
Capstone Deliverables
Submit:
- service brief and frozen evaluation set;
- architecture/data-flow/threat diagram;
- pinned artifact/config manifest and license review;
- protected gateway and synthetic API tests;
- benchmark/load/recovery report with raw evidence;
- privacy, retention, logging, backup, and incident rules;
- dated local/cloud/hybrid cost model;
- operations/handoff/retirement runbooks;
- acceptance sign-off or documented rejection.
Completion Rubric
- Complete: task quality, provenance, security, safe networking, load envelope, recovery, observability, cost, and second-operator handoff all pass.
- Needs revision: inference works but one of identity, overload, recovery, evidence, or ownership remains unproven.
- Not complete: raw ports are exposed, sensitive data is mishandled, protections are bypassed, or capacity/savings are claimed beyond tests.
Sources
- llama.cpp HTTP server documentation
- Ollama API documentation
- OWASP API Security Top 10
- NIST AI Risk Management Framework
- vLLM serving benchmark CLI
Key takeaway: A local inference server is production-ready only when a pinned model passes the task and the surrounding service proves identity, limits, recovery, observability, cost, and ownership.