Every webhook sender retries. A timeout, a deploy, a load balancer dropping a connection halfway through your response — the sender cannot distinguish those from a genuine failure, so it sends again. Your endpoint returned success and processed the event, and it is about to process it a second time.
The usual advice is to make your handler idempotent, which is true and unhelpfully abstract. Idempotency is not a property you add to a function; it is a record you keep, in a place that survives a crash, written under a constraint the database enforces.
The record, and where the constraint goes
Every delivery carries an event identifier that is stable across retries. Store it with a unique constraint, and let the insert be the thing that decides whether this is a first delivery or a repeat. A read-then-write check does not work: two concurrent retries both read nothing, both proceed, and you have sent two invitation emails from code that looked like it was guarded.
Write the identifier in the same transaction as the effect it protects. If the identifier commits and the effect does not, you have permanently dropped the event; if the effect commits and the identifier does not, the retry duplicates it. One transaction removes both, and the effects that cannot participate in your transaction are exactly the ones that need more care.
- Unique index on the event identifier — the database decides, not your code
- Insert and effect in one transaction, or you have moved the race rather than fixed it
- Return success on a duplicate; a conflict status invites the sender to retry harder
- Keep the records long enough to outlive the sender's retry schedule, then expire them
Idempotency is not a property of your handler. It is a unique constraint your handler cannot get around.
Side effects that leave the transaction
Sending an email, charging a card, calling somebody else's API — these do not roll back. The pattern that works is to make the durable record inside the transaction and the outside call after it, driven by that record. Write a row saying this email should be sent, commit, and let a worker send it and mark it done. A crash between the two produces a delayed email, which is a much better failure than a duplicate charge.
Where the downstream service supports an idempotency key of its own, derive it from the event identifier rather than generating a fresh one per attempt. That way your retry and their deduplication agree about what constitutes the same operation.
Ordering is a separate problem
Deduplication stops you applying the same event twice. It does nothing about applying two different events in the wrong order, which is just as common: an update and the deletion that followed it can arrive in either sequence, and processing them as they land leaves you with a resurrected record.
Use a version or timestamp from the sender and refuse to apply an event older than the state you already hold. If the payload carries no such field, fetch the current state from the source rather than trusting the ordering of deliveries. Neither is much code, and both remove a class of bug that is almost impossible to reproduce once it is in production.
- Compare a sender-supplied version before applying; drop what is stale
- Treat the payload as a notification and re-read the source where correctness matters
- Verify the signature before touching the body, including on retries
- Reject deliveries whose signed timestamp is old, so a captured one cannot be replayed
Everything here, already built
Sign-in, enterprise SSO, directory provisioning, roles and an audit trail behind one API. Start with the quickstart and have a working sign-in this afternoon.