2026-09-17 · 8 min read
- webhooks
- idempotency
- reliability
Webhook delivery is normally at least once. A provider retries when an acknowledgement is missing, and that can happen even after the receiver committed the first attempt. Duplicates and delayed delivery are therefore expected inputs.
A robust receiver authenticates the exact request, stores a durable inbox record, acknowledges within the provider's deadline, and processes effects under a separate retry policy.
Verify before parsing or deduplicating
Capture the request body as bytes and verify the provider's signature using its documented construction, timestamp tolerance, key selection, and replay rules. JSON parsing or reserialization before verification can change authenticated bytes.
- Limit body size before expensive processing.
- Use the provider's stable event identity when available.
- Keep signature failures out of the durable processing queue.
- Redact secrets and personal data from diagnostics.
Claim each logical event atomically
Insert the provider, account scope, and event ID under a unique constraint. Concurrent deliveries should converge on one inbox item while each HTTP attempt remains observable separately.
Store processing status and effect checkpoints so a worker can distinguish never started, in progress, completed, retryable, and terminal states. Returning success for an already accepted event stops needless provider retries.
Handle ordering at the business object
Receipt order is not creation order. Prefer retrieving current provider state when an event is a notification. Otherwise, use a trusted sequence, object version, or event-time policy to prevent stale updates from overwriting current state.
Partition ordered work by the affected object when possible. Globally serializing unrelated webhook events reduces throughput without fixing missing or delayed events.
Exercise acknowledgement and commit gaps
Use Flashman's hash tool for exact-byte fixtures, JSON formatter for synthetic events, timestamp converter for event and receipt times, UUID generator for local attempts, and diff for processing histories.
Test simultaneous duplicates, delayed older events, response loss after commit, timeout before commit, worker crashes, malformed signatures, key rotation, replay windows, poison events, provider redelivery limits, manual replay, downstream idempotency expiry, and account isolation.