Rate limiting strategies

Four shapes of limiter

Every limiter answers the same question — may this request proceed — and they differ in how they treat a burst. Knowing which one you are behind explains why a rate that seemed safe still returned 429.

What to do with a 429

A 429 is information, not a failure. It carries a Retry-After header saying how long to wait, and honouring it is both the fastest way through and the difference between a client that recovers and one that makes things worse.

1. 1Read Retry-After and wait exactly that long. It is not a suggestion, and guessing shorter simply produces another 429.
2. 2Where no header is present, back off exponentially from a small base, with a ceiling. Doubling from one second to a thirty-second cap covers almost every real recovery.
3. 3Add jitter. Without it, every worker that was rate limited at the same moment retries at the same moment, which reproduces the burst that caused the limit.
4. 4Cap total attempts and surface a failure rather than retrying forever. A request retried for an hour has usually stopped being useful.
5. 5Never retry a non-idempotent write without an idempotency key. A 429 means the request was rejected, but a 500 or a timeout does not, and the same retry code will meet both.

1async function withRetry(fn, attempts = 5) {
2  let wait = 1000;
3 
4  for (let i = 0; i < attempts; i++) {
5    const response = await fn();
6    if (response.status !== 429) return response;
7 
8    // Retry-After varsa ona uy; yoksa geri cekilerek bekle.
9    const header = response.headers.get("Retry-After");
10    const delay = header ? Number(header) * 1000 : wait;
11 
12    // Jitter olmadan tum isciler ayni anda tekrar dener.
13    await new Promise((r) => setTimeout(r, delay + Math.random() * 250));
14    wait = Math.min(wait * 2, 30000);
15  }
16 
17  throw new Error("rate limited");
18}

Not hitting the limit in the first place

Retrying well is a recovery mechanism. The requests that never happen are cheaper than the ones that are retried politely, and most integrations that hit limits are doing something avoidable.

Bulk reads are the usual culprit: a nightly job that reads every directory user for every tenant, sequentially, as fast as the client library allows. Paginating with a sensible limit, spacing tenants out, and reacting to events rather than polling removes most of that traffic entirely.

1. 1Prefer events to polling. A webhook that tells you what changed replaces a job that reads everything to find out.
2. 2Use the largest page size the endpoint allows. One hundred records in one request is not the same cost as one hundred requests.
3. 3Cache what does not change often — role definitions, organization names — with a short lifetime rather than reading them per request.
4. 4Spread scheduled work. Every tenant syncing at midnight is a burst you created.
5. 5Run bulk work with a concurrency limit rather than unbounded parallelism, so a slow response does not cause a pile-up.

Limiting your own inbound traffic

The other direction matters just as much. A customer's directory doing a full resync sends your webhook endpoint tens of thousands of deliveries in a short period, and an endpoint that processes them synchronously will time out — which pauses the endpoint, which loses events you cared about.

The shape that survives is to acknowledge quickly and process asynchronously: verify the signature, write the delivery to a queue, return 200, and do the work elsewhere. Then limit by tenant in the worker, so one customer's resync does not starve everybody else's ordinary traffic.

This is also where a concurrency limit earns its place over a rate limit. Your database has a finite number of connections, and capping in-flight work protects it in a way that capping requests per second does not.

Related