A local inference API separates applications from the model runtime, but it also creates a service boundary that needs authentication, authorization, validation, limits, observability, and lifecycle management. Begin on localhost. Do not expose a raw model port to a LAN or internet merely because a demo request works.

// concept

Define the API Contract

Choose stable request fields: model alias mapped server-side, messages/input, allowed parameters, output schema, maximum input/output tokens, request ID, and timeout. Return consistent errors for validation, overload, timeout, policy refusal, and internal failure.

Do not let untrusted clients choose arbitrary local file paths, model repositories, adapters, tools, or templates. Keep an allowlist and map public aliases to pinned internal revisions. Validate content type and body size before inference.

Ollama and llama.cpp offer HTTP APIs; llama.cpp can provide compatible routes. Compatibility does not mean identical semantics. Test streaming, token counting, tool/schema behavior, errors, and cancellation against the exact version.

// concept

Add a Protective Gateway

Place application controls in front of the runtime:

authenticated identities and per-tenant authorization;
TLS for network traffic and secret management;
request/body/token/rate/concurrency limits;
bounded queue, timeouts, cancellation, and retry policy;
input/output policy and schema validation;
audit-safe request IDs and metrics without raw sensitive prompts;
health/readiness endpoints and graceful shutdown;
versioned model routing and rollback.

Do not embed a shared secret in public browser JavaScript. The browser should call an authenticated application backend. Rotate credentials and avoid logging authorization headers.

// concept

Start Local and Test

Keep the runtime bound to loopback and make a synthetic request. An Ollama API pattern is:

// powershell9 lines

$body = @{
  model = '<approved-model:variant>'
  prompt = 'Return JSON with status set to ok.'
  stream = $false
} | ConvertTo-Json

Invoke-RestMethod `
  -Uri 'http://localhost:11434/api/generate' `
  -Method Post -ContentType 'application/json' -Body $body

Pin the model internally, validate output, and record latency/errors. The model’s response is untrusted data; escape it in HTML, validate JSON, and never execute generated commands automatically.

// concept

Reliability and Observability

Track accepted/rejected requests, queue time, TTFT, total latency, token counts, model revision, error classes, memory, restarts, and health. Avoid raw prompt logs by default; if approved debugging requires content, restrict access, redact, expire, and document it.

Readiness should fail until the intended model is usable. Liveness should detect a stuck process without triggering restart loops during overload. Deploy changes with a small synthetic smoke set and rollback.

// pakistan_angle

Pakistan Angle

For an office LAN, treat every device as untrusted until authenticated. Shared Wi-Fi, contractor laptops, or misconfigured routers make “inside the office” a weak boundary. Use host firewalls and an approved secure-access solution rather than router port forwarding.

If power or internet changes, clients need bounded retry/backoff and idempotency for jobs. Do not claim offline resilience until shutdown, restart, queue recovery, and corrupted/incomplete job handling are tested.

// hands_on

Hands-On Exercise

Design a localhost-only API proof with one pinned model and synthetic inputs. Add a contract, validation, output schema, token/body limits, timeout, queue cap, request IDs, redacted metrics, health/readiness, and ten tests including overload and malformed inputs. Document the network boundary; do not expose it publicly.

// completion_rubric

Completion Rubric

3 grading bands

Complete
runtime is isolated behind a validated authenticated contract with limits, observability, pinned versions, and rollback tests.
Needs revision
the API works but browser secrets, logs, overload, cancellation, or readiness are weak.
Not complete
an unauthenticated raw model endpoint is exposed or generated output is trusted/executed.

// sources

Sources

4 official sources — check every claim yourself