Calculators

AI and automation

API Rate-Limit and Throughput Planner

Plan safe AI API throughput from RPM, TPM, request size, retries, latency, concurrency, and batching.

When to use this

Use it before queues, workers, or agents hit throttling

This planner is for batch classifiers, AI agents, RAG answers, lead enrichment, background summarizers, and workflow tools where request limits, token limits, and latency interact.

Default result

The server-rendered header sample has a safe throughput of 48/min and status: Within safe throughput.

Estimate safe throughput

Start with a traffic preset, then paste the current RPM and TPM limits from your API account.

Planning rule

Batching reduces API requests per minute, but tokens per task still count against TPM. Conservative planning uses max output tokens, not only average output tokens.

Reserved tokens

2,500 tokens per API request

2,500 tokens per task after max output reservation.

Bottleneck

RPM and TPM

Raw limit: 60/min before safety buffer.

Worker pool

6 workers required

Current workers can process 60/min at the selected latency.

Worked example: 60 RPM and 150,000 TPM header sample

The default scenario uses 60 RPM and 150,000 TPM. Each task reserves 1,800 input tokens plus 700 max output tokens, so each task reserves 2,500 tokens.

At batch size 1, RPM allows 60 tasks/min. TPM allows floor(150,000 / 2,500) = 60 tasks/min. The raw limit is therefore 60 tasks/min.

With a 20% safety buffer, safe API throughput is 60 x 0.80 = 48 tasks/min. Eight workers at 8 seconds average latency can process 8 x 60 / 8 = 60 requests/min, or 60 tasks/min at batch size 1.

The target is 40 tasks/min with 5% retry overhead, so adjusted demand is 42 tasks/min and 42 x 2,500 = 105,000 TPM. That uses 70% of RPM and 70% of TPM, which fits under the 48 tasks/min buffered throughput.

How the API throughput planner works

The planner compares three ceilings: tasks allowed by RPM, tasks allowed by TPM, and tasks your worker pool can process at the selected latency. It then applies a safety buffer to provider limits and inflates the target by retry overhead.

RPM capacity equals requests per minute times batch size. TPM capacity equals tokens per minute divided by tokens reserved per task. Worker capacity equals concurrency times 60 divided by average latency, then multiplied by batch size.

How to read RPM and TPM from headers

Use response headers or dashboard limits from the provider. For OpenAI-style headers, request and token limits can be separate, and whichever limit is exhausted first throttles the workflow.

When batching improves throughput

Batching helps when RPM is the bottleneck and each task is small enough that TPM still has headroom. It can hurt when responses must be immediate, when batch parsing is fragile, or when one failed batch creates too much retry work.

What this estimate excludes

The result excludes daily limits, per-user caps, model-family shared pools, image or audio-specific quotas, regional limits, serverless cold starts, queue overhead, provider reset windows, and downstream database or webhook bottlenecks.

What to do next

Log real request tokens, output tokens, latency, retry count, and headers in a staging run. Then rerun the planner with observed p95 latency and a larger safety buffer before opening traffic to users.

Reference data used by the defaults

Topic Reference value Source Date Note
Rate-limit dimensions OpenAI documents requests per minute, requests per day, tokens per minute, tokens per day, images per minute, and audio minutes as rate-limit metrics. OpenAI rate limit guide As of June 20, 2026 This calculator focuses on RPM and TPM because they are the common synchronous text bottlenecks.
Rate-limit headers OpenAI example headers include x-ratelimit-limit-requests: 60 and x-ratelimit-limit-tokens: 150000. OpenAI rate limit guide As of June 20, 2026 Use your actual response headers or dashboard limits, not the sample values, for production planning.
Retries and rate limits Unsuccessful requests contribute to the per-minute limit. OpenAI rate limit guide As of June 20, 2026 The retry percentage input inflates RPM and TPM demand to avoid optimistic throughput.
Max token reservation OpenAI says rate limit calculation uses the maximum of max_tokens and the estimated tokens from request character count. OpenAI rate limit guide As of June 20, 2026 The calculator uses max output tokens per task as a conservative TPM planning input.
Batching for RPM-bound workloads OpenAI notes that batching can increase throughput when requests per minute are the bottleneck and tokens per minute capacity remains available. OpenAI rate limit guide As of June 20, 2026 Batching reduces request count but does not reduce token volume, and it can increase latency or response parsing complexity.

Provider limits, model families, usage tiers, reset windows, and throttling behavior can change. Use current account data before load testing.

FAQ

Is this only for OpenAI?

No. The defaults and references use OpenAI because the public guide documents RPM, TPM, headers, retries, and batching clearly. You can enter any provider limits that use request and token quotas.

Should I use expected output tokens or max output tokens?

Use max output tokens for conservative rate-limit planning when the provider reserves against max_tokens. Use expected output tokens only after usage logs prove the provider accounting is lower for your endpoint.

Why does batching help only sometimes?

Batching reduces requests per minute, but the same tasks still consume tokens. It helps when RPM is the bottleneck and TPM has room. It does not help when TPM or latency is already the limiting factor.

How should retries be modeled?

Retries consume more request capacity and can also consume token capacity. Enter an overhead percentage for failed calls, timeouts, duplicate work, or backoff retries so the plan is not sized at the edge.

What does worker concurrency mean?

It is the number of in-flight API requests your app, queue, serverless function, or worker pool can sustain. More workers do not bypass provider limits, but too few workers can keep you below the allowed throughput.

What safety buffer should I use?

Start with 15% to 25% for production workflows. Use more when user input, retrieved context, tool schemas, retries, or output length vary widely.

Decision path

What to do next

Request size

Count tokens before scaling

Use token and context planning to replace rough input and output token estimates.

 API budget

Estimate the monthly API bill

Throughput that fits limits still needs a token budget and model pricing check.

 Workflow path

Plan queue and worker design

Connect rate limits with automation design, retries, queues, and provider choice.