Safe Scheduler — Idempotent Research Jobs Automate Karo

A scheduler should automate evidence capture, not multiply mistakes. Jobs can overlap after a slow network call, rerun after a restart, or fire twice during deployment. Design every job as if duplicate delivery is normal. The goal is idempotency: repeating the same logical run does not create contradictory analytical events.

Derive a deterministic run_key from job name, scheduled UTC window, config hash, and dataset source. Before work begins, insert a run record with a unique constraint on that key. If it already exists in SUCCEEDED state, skip. If it is RUNNING but its lease is stale, record recovery and resume from immutable inputs. Never delete the first attempt.

Use APScheduler with an explicit timezone, max_instances=1, coalescing rules, and a misfire grace period chosen from the data freshness requirement. These options reduce overlap, but the database uniqueness rule remains the real protection. A second process or redeployment can bypass in-memory scheduling controls.

The job flow is:

1. Acquire the run lease.
2. Resolve and hash configuration.
3. Fetch or load raw data with a bounded deadline.
4. Validate and persist normalized records.
5. Run deterministic research and paper simulation.
6. Append metrics and report artifact hashes.
7. Mark success only after every required artifact exists.

On failure, append a sanitized error category and mark FAILED; do not pretend an old report is current. Retry only errors classified transient, with a capped attempt count and jitter. Validation, forbidden-host, and schema-version errors require operator review.

Clock discipline

Schedule and store in UTC. The display may show Pakistan time. Avoid using “now” repeatedly inside calculations; capture one run timestamp and pass it through. In tests, inject a clock so misfire and stale-lease behavior is deterministic.

🇵🇰 Pakistan Angle

Power cuts and unstable connectivity make recovery a first-class feature. A missed job should be logged, not backfilled with invented snapshots. After connectivity returns, capture a new observation and preserve the gap. Set schedules that respect limited data bundles and use cached fixtures for practice.

Hands-On Exercise

Implement a scheduled fixture job and a runs table with a unique run_key. Trigger it twice concurrently; prove one succeeds and one reports duplicate without duplicated events. Simulate interruption after normalization, expire the lease, and recover using the same raw hash. Show the gap rather than manufacturing a missed quote.

Completion Rubric

• The logical run key is deterministic and database-enforced.
• Overlap, misfire, retry, lease, and recovery policies are explicit.
• Non-transient validation errors are not retried.
• Missed observations remain visible as gaps.
• Success requires all declared artifacts and hashes.

Sources

• APScheduler user guide
• SQLite unique constraints
• Python timezone-aware datetime guidance

Key takeaway: Scheduler settings reduce duplicate work, while idempotent storage and immutable inputs make retries trustworthy in practice across tested restarts and controlled deployments.