Module 09 · Deploying the Research Service — Read-Only and Measured
Advanced Research — Failure Injection and Robustness
Open lesson + course map
On this lesson
Course outline
Module 1 · Market Systems and Safety — Pehle Boundaries Samjho
Module 2 · Python Bot Architecture — Ek Professional Bot Ka Skeleton
Module 3 · Market Data Pipeline — Read-Only Evidence Safely Fetch Karo
Module 4 · AI Research Engine — Extraction Se Human Review Tak
Module 5 · Strategy Research — Hypothesis Se Paper Test Tak
Module 6 · Paper Execution Engine — Synthetic Fills Only
Module 7 · Risk Controls — Estimation Error and Paper Limits
Module 8 · Database and Monitoring — Audit Logging and Model Evaluation
Module 9 · Deploying the Research Service — Read-Only and Measured
Robustness is evidence that the system fails safely under realistic faults. Build a failure catalogue across network, data, model, clock, database, disk, scheduler, and operator layers. For each fault, define expected detection, containment, recovery, and audit evidence before injecting it.
Network faults include timeout, DNS failure, 429, partial pagination, redirect escape, and truncated body. Data faults include schema drift, duplicated IDs, future timestamps, missing quotes, changed resolution rule, and poisoned text. Model faults include invalid JSON, unknown source IDs, hallucinated quotations, excessive output, and prompt injection.
Operational faults include concurrent duplicate runs, clock skew, process termination between stages, SQLite lock, disk full, corrupt projection, failed backup, SMTP outage, and expired human approval. Use fixtures, mocks, temporary filesystems, and a local fake server; never attack an external provider.
Record a robustness scorecard with fault ID, injected condition, expected control, observed result, detection time, recovery steps, evidence link, and residual risk. Passing means safe behavior—not uninterrupted output. An abstention, quarantined run, or degraded dashboard may be the correct result.
Run repeated and combined faults. A timeout plus stale cache may expose behavior absent in isolated tests. Randomize within bounded seeds and store the seed for replay. Keep a stable core suite in continuous integration and a slower recovery drill before releases.
Measure false recovery: cases where the service claims healthy while data is stale, projection mismatched, or alert delivery failed. These are more dangerous than visible downtime. Add invariants that block report freshness until sealing, reconciliation, and safety checks pass.
Assign each fault a blast radius and containment boundary. A malformed document should quarantine one record; a parser schema failure should fail the run; a broken hash chain should quarantine the portfolio; a safety-manifest failure should stop the service release. Verify faults do not cascade unnecessarily while critical integrity faults cannot be isolated away and ignored.
After each drill, append—not overwrite—a corrective-action record with owner, test added, expected completion evidence, and residual limitation. Rerun both the fault and neighboring normal cases to ensure the fix did not merely change the failure mode.
// pakistan_angle
Pakistan Angle
Include power loss and intermittent connectivity as normal scenarios, not exotic chaos. The goal is truthful gaps and clean recovery on modest hardware. Do not use proxies or access circumvention to simulate availability.
// hands_on
Hands-On Exercise
Implement twelve faults spanning the categories above, including one combined outage. Run the scorecard, fix one unsafe recovery, and rerun from the same seed. Demonstrate that no fault can create an unaudited synthetic fill or current-looking stale report.
// completion_rubric
Completion Rubric
5 checks — tick as you verify
// sources
Sources
3 official sources — check every claim yourself