Delivery guarantees are usually discussed as a single property, and consumers read at-least-once as a caveat about occasional duplicates. It is better read as a description of the normal case: retries happen constantly, from timeouts, deploys and dropped connections, and any handler that is correct only when each event arrives exactly once is a handler that is intermittently wrong.
Deduplication handles the repeat. It does nothing about sequence, and sequence is the failure that survives all the obvious defences: an update and the deletion that followed it can arrive in either order, and applying them as they land leaves a record that was deleted and then quietly restored.
Per-entity ordering is achievable; global ordering is not
A total order across every event a system emits would require a single serialisation point that every producer passes through, which is exactly the bottleneck distributed systems are built to avoid. Senders that promise it are either very small or very slow, so the practical guarantee is narrower: events concerning one entity are ordered relative to each other, and events concerning different entities are not comparable.
That narrower promise is enough, because handlers almost never need to compare a user event with an unrelated organisation event. Design the consumer to care about order only within an entity, and process different entities concurrently. The moment your code depends on the relative order of two events about different things, you have taken a dependency the sender did not offer.
Order within one entity is a guarantee you can build on. Order across the whole stream is a guarantee nobody can give you.
The event is a notification, not the state
The most reliable consumers treat a delivery as a hint that something changed, then read the current state from the source before acting. Ordering problems disappear because you are never assembling state from a sequence of fragments; you are refreshing a copy. Duplicates become harmless for the same reason, and a missed event is repaired by the next one that touches the same entity.
The trade is a read per event, and it is worth it wherever correctness matters more than throughput. Where volume makes the read impractical, apply the payload but carry a version: the sender includes a monotonic value per entity, you store the last one applied, and you drop anything not newer. That comparison is a few lines and removes an entire class of bug that is nearly impossible to reproduce.
- Re-read the source where correctness matters more than the cost of a lookup
- Where you apply the payload directly, compare a per-entity version and drop stale events
- Never derive a running total by adding up deliveries; recompute from the source
- Deletion must win over an older update, whatever order the two arrive in
Noticing a gap
At-least-once says nothing about eventually. An endpoint that returns errors for an hour, or a retry schedule that exhausts, produces events that never arrive at all, and a consumer that only processes what it receives has no way to know. The state drifts, and the discovery happens weeks later when a customer notices somebody still has access they should not.
Two mechanisms cover it, and mature systems run both. A per-entity sequence number lets a consumer detect that it jumped from one value to another with something missing in between, which turns a silent gap into an alert. And a periodic reconciliation sweep — list the current state from the source, compare with what you hold, repair the differences — catches everything the stream lost, including the events that were never emitted because of a bug on the sending side.
The backfill boundary
Every consumer eventually needs to load history and then continue from the live stream, whether at first setup or after an outage. The join between those two is where duplicates and gaps are manufactured. Take the snapshot first and start consuming from a position recorded before the snapshot began, so the overlap replays events you have already applied rather than skipping ones you have not.
That is only safe because the handler is idempotent and version-aware, which is the point: those two properties are not defensive extras, they are what makes ordinary operations like backfilling, replaying and re-pointing a consumer routine instead of frightening. A system where replay is safe is a system you can operate. One where it is not will accumulate drift nobody dares to fix.
- Record the stream position before taking the snapshot, and accept the overlap
- Make replay of an arbitrary time range a supported operation, not an emergency script
- Alert on sequence gaps per entity rather than on aggregate delivery counts
- Reconcile on a schedule; it is the only thing that catches events never sent
Everything here, already built
Sign-in, enterprise SSO, directory provisioning, roles and an audit trail behind one API. Start with the quickstart and have a working sign-in this afternoon.