2026-08-27 · 8 min read
- cron
- scheduling
- reliability
Cron and cloud schedulers translate wall-clock time into dispatch decisions. Wall clocks are synchronized rather than perfectly continuous: they can be slewed gradually, stepped after a large offset, repeated during a backward correction, or skipped after suspend and resume.
Scheduler behavior during these events differs by implementation. Some avoid duplicate fixed-time jobs after small backward changes, some catch up missed windows, and distributed control planes may redeliver independently of host time.
Separate wall time from elapsed time
Wall time answers which calendar window a job belongs to. A monotonic clock measures elapsed duration without moving backward when civil time is corrected. Use wall time for schedule identity and monotonic time for timeouts, latency, and runtime measurement where the platform provides it.
- Store timestamps with UTC offsets or as unambiguous epoch values.
- Record the scheduler's intended window as well as actual start time.
- Capture host time-service and VM lifecycle events.
- Do not infer elapsed duration by subtracting unstable wall-clock samples.
Use a deterministic execution key
Construct a key from stable job identity and the intended schedule window, such as billing-summary:2026-08-27. Claim that key atomically in durable storage before performing side effects. A retry or duplicate scheduler dispatch then observes the existing claim.
A random correlation ID identifies one attempt but cannot deduplicate multiple attempts. Keep both values: deterministic execution key for correctness and a UUID for tracing.
Design recovery states explicitly
A claimed key must distinguish running, completed, retryable failure, and permanently failed work. Otherwise a process crash after claiming but before completing can block recovery forever.
- Use leases or transactional outbox patterns where appropriate.
- Make downstream operations idempotent with their own stable keys.
- Define whether missed windows are skipped, merged, or backfilled.
- Alert on stale running records and unexpected duplicate attempts.
Rehearse time anomalies
Test duplicate dispatch directly instead of changing a production host clock. Invoke the same logical window twice, run two workers concurrently, and simulate a crash around each state transition. Also verify platform behavior after controller failover and VM resume in a disposable environment.
Use Flashman's cron tool to preview intended windows, timestamp converter to normalize logs, diff tool for scheduler config, UUID generator for attempt traces, and JSON formatter for job records. The durable execution key—not the preview—is what protects production side effects.