2026-09-13 · 6 min read · Rahul Chitturi
- cron
- distributed-systems
- reliability
A distributed scheduler often elects one leader to dispatch each due job. During a pause, network partition, lease expiry, or rolling deployment, an old leader may continue briefly after a new leader starts. Both can enqueue the same logical run.
Leader election reduces overlap but cannot by itself guarantee exactly-once effects across a queue, worker, database, and external API.
Give every scheduled occurrence an identity
Derive a stable occurrence key from the schedule, job identity, intended fire time, and timezone or schedule version. Persist dispatch claims atomically so two leaders agree whether that occurrence already exists.
- Use UTC instants after applying calendar semantics.
- Version schedules when expressions or timezones change.
- Keep retry attempt IDs separate from occurrence IDs.
- Record lease generation or fencing tokens.
Fence stale leaders and workers
A lease timestamp alone does not stop a paused process from resuming. Pass a monotonically increasing generation to writes and reject work from older generations where the storage system can enforce that comparison.
Make side effects idempotent with unique constraints, operation keys, conditional updates, or downstream idempotency support. Acknowledgement loss can redeliver work even when leadership is perfect.
Exercise the failover window
Use cron helper to confirm intended fire times, timestamp converter to align lease events, UUID generator for attempt IDs, and diff for sanitized dispatcher histories.
Test process pauses, clock skew, network partitions, lease renewal delay, rolling deploys, queue acknowledgement loss, retries, daylight saving changes, backfills, manual runs, and side effects after timeout.