Webhook reliability

Verify before you parse

Your endpoint is on the public internet, so anybody can POST to it. The signature is what distinguishes a real delivery from a request somebody crafted, and it must be checked against the raw body — parsing to JSON first and re-serialising changes the bytes and breaks the comparison.

The signature also carries a timestamp, which is what prevents an old, genuine delivery from being replayed. Reject anything older than a few minutes.

1import { Paycux } from "@paycux/node";
2 
3const paycux = new Paycux(process.env.PAYCUX_API_KEY);
4 
5export async function POST(request: Request) {
6  // ham govde — once metin olarak okunur, JSON'a cevrilmez
7  const payload = await request.text();
8  const signature = request.headers.get("paycux-signature");
9 
10  let event;
11  try {
12    event = paycux.webhooks.verify({
13      payload,
14      signature,
15      secret: process.env.PAYCUX_WEBHOOK_SECRET,
16      toleranceSeconds: 300,
17    });
18  } catch {
19    return new Response("invalid signature", { status: 401 });
20  }
21 
22  await enqueue(event);
23  return new Response("ok", { status: 200 });
24}

Assume every delivery arrives twice

Retries are normal, not exceptional. A delivery times out after your handler has already committed, a network hiccup interrupts the response, a queue redelivers — and the same event arrives again. If your handler is not idempotent, the visible result is a duplicate invitation email or a doubled group membership.

Record the event ID before doing the work and check it first. A unique constraint on that column is the whole mechanism, and it survives concurrent deliveries in a way that a read-then-write check does not.

1export async function claimEvent(eventId: string) {
2  try {
3    // benzersiz kisit ayni olayin iki kez islenmesini engeller
4    await db.processedEvent.create({ data: { id: eventId } });
5    return true;
6  } catch (error) {
7    if (error.code === "P2002") return false; // zaten islenmis
8    throw error;
9  }
10}

Acknowledge fast, work later

Return a 200 as soon as you have verified and stored the event. Everything else — sending mail, calling other services, rebuilding a search index — belongs on a queue.

This is not only about latency. A handler that does its work inline will eventually be slower than the delivery timeout, at which point every event is retried, which doubles the load, which makes it slower still. Retry storms are self-inflicted and easy to avoid.

Ordering is not guaranteed

Two events emitted a second apart can arrive in either order, and after a retry they routinely do. A handler that applies a diff — add this group, remove that one — produces the wrong end state when they swap.

Treat each event as a statement of the desired end state and upsert the whole record. Where that is impossible, compare the event's timestamp against what you last applied and drop anything older.

1export async function applyUserEvent(event) {
2  const existing = await db.user.findUnique({
3    where: { directoryUserId: event.data.id },
4  });
5 
6  // eski bir teslimat yeni durumu ezmesin
7  if (existing && existing.syncedAt > new Date(event.createdAt)) return;
8 
9  await db.user.upsert({
10    where: { directoryUserId: event.data.id },
11    create: toUser(event.data, event.createdAt),
12    update: toUser(event.data, event.createdAt),
13  });
14}

When you drop one anyway

Eventually you will lose an event: your endpoint was down longer than the retry window, a deploy dropped in-flight requests, a bug rejected valid deliveries for an hour. Plan for it rather than hoping.

• Keep the raw payload of every delivery you accept, for long enough to replay it. Storage is cheaper than reconstructing state from a customer's directory.
• Build a resync path — a full read of current state, applied idempotently. If your handler is written correctly this costs almost nothing extra.
• Alert on the absence of events, not only on errors. A silent endpoint looks identical to a healthy one from the inside.
• Log the event ID with every handled event so a support question can be answered from your own logs.

What to test

1. 1A valid delivery, then the same delivery again. The second must change nothing.
2. 2A delivery with a tampered body and a valid-looking signature. It must be rejected.
3. 3A delivery with a timestamp an hour old. It must be rejected.
4. 4Two related events applied in reverse order. The end state must be correct.
5. 5Your endpoint returning 500, then succeeding on retry. Nothing may be applied twice.

Related