2026-09-17 · 6 min read · Rahul Chitturi
- webhooks
- idempotency
- reliability
Webhook providers retry when a response is late, lost, or unsuccessful. The first handler attempt may have committed even when the provider never received its acknowledgement, so a repeated delivery is normal distributed-system behavior rather than proof of a sender bug.
Delivery order can also differ from event creation order. Processing every valid signature immediately can let an older update overwrite newer state.
Separate event identity from delivery attempts
Persist the provider's stable event ID, event type, subject, creation time, and a bounded digest or storage reference for the verified body. Record each delivery attempt separately with receipt time and outcome.
- Verify the signature over the exact received bytes first.
- Claim the event ID atomically before applying effects.
- Return success for an already completed event.
- Keep internal retry IDs distinct from provider delivery IDs.
Protect state from stale events
Prefer fetching current authoritative state when a webhook is only a change notification. If payloads are applied directly, compare provider sequence numbers, object versions, or trusted event times under a documented ordering contract.
Do not hold an inbound HTTP request open for every downstream action. A durable inbox can acknowledge promptly and let workers retry with idempotency keys and observable failure states.
Build a replay-safe fixture
Use the JSON formatter for synthetic event bodies, hash tool for exact-byte fixture labels, timestamp converter for event and receipt timelines, and UUID generator for local attempt IDs.
Test duplicate simultaneous deliveries, delayed older events, acknowledgement loss, malformed signatures, key rotation, timeouts before and after commit, poison events, worker crashes, manual replay, and downstream idempotency expiry.