2026-09-13 · 8 min read
- cron
- distributed-systems
- reliability
A scheduler running on several instances needs to decide which process dispatches each occurrence. Leased leadership is common, but pauses, partitions, delayed renewals, and deployments create windows where old and new leaders can both act.
The system should target at-least-once dispatch with idempotent effects instead of claiming exactly-once execution across independent storage, queue, worker, and external-service boundaries.
Model occurrences independently from attempts
Create a stable occurrence identity from the job, schedule version, intended fire instant, and relevant calendar policy. A retry or redelivery gets a separate attempt identity while retaining the same occurrence identity.
- Persist timezone and daylight-saving policy.
- Version schedule edits and deployment ownership.
- Distinguish manual runs from scheduled occurrences.
- Define backfill and misfire identities explicitly.
Use durable claims and fencing
Atomically insert or claim an occurrence in authoritative storage before dispatch. A unique constraint can prevent two leaders from creating distinct work for the same occurrence.
Issue a monotonically increasing leadership generation and require state-changing writes to reject stale generations where possible. A timestamp lease alone cannot stop a paused old process after it resumes.
Make every effect idempotent
Queues can redeliver after acknowledgement loss, workers can time out after committing, and external APIs can succeed before a connection fails. Carry an idempotency key through each boundary and record outcomes durably.
- Use unique constraints or conditional updates.
- Prefer transactional outbox dispatch for database-backed changes.
- Do not hold leadership while waiting on slow external work.
- Bound retries and surface terminal occurrences.
Run adversarial failover tests
Use Flashman's cron helper for intended schedules, timestamp converter for lease timelines, UUID generator for attempt IDs, diff for sanitized histories, and hash for stable public fixtures.
Test process suspension, lease expiry, clock skew, partitions, renewal delay, rolling deploys, queue redelivery, acknowledgement loss, worker crashes, timeout after commit, daylight-saving transitions, schedule edits, backfills, and manual retries.