Last updated: 2026-06-22

LLM rate-limit planning explained

LLM rate-limit planning is the process of determining the sustainable request rate, and the in-flight concurrency it implies, that a provider's API limits allow for a given workload. Hosted inference APIs cap throughput on two independent axes: tokens per minute (TPM), the number of tokens — input plus output — the account may process each minute, and requests per minute (RPM), the number of API calls allowed each minute regardless of their size. Whichever cap is reached first sets the ceiling.

Which limit binds

A workload with an average token cost per call translates the TPM cap into an equivalent request cap. Dividing the per-minute token budget by the tokens consumed per request gives the requests per minute the token limit alone permits:

where $L_t$ is the TPM limit and $n$ is tokens per request. The sustainable rate is the smaller of this token-derived cap and the explicit RPM cap $L_r$:

If $R_t < L_r$ the token limit binds; if $R_t \ge L_r$ the request limit binds. Token-heavy work (long context, large completions) tends to be token-bound; high-frequency small calls tend to be request-bound.

Concurrency from Little's law

The sustainable rate per second is $R_s = R / 60$. Little's law, a queueing result stating that the average number of items in a system equals the arrival rate times the average time each spends in it, converts that rate and the mean request latency $t_L$ into the concurrency $C$ that must run in parallel to keep the pipe full:

Worked example

Consider a TPM limit of $L_t = 2{,}000{,}000$, an RPM limit of $L_r = 10{,}000$, and an average of n = 2{,}000 tokens per request. The token-derived cap is

Since 1{,}000 < 10{,}000, the token limit binds: throughput is capped at 1,000 requests per minute, not the headline 10,000. At an average latency of $t_L = 20$ seconds, the per-second rate is $R_s = 1{,}000 / 60 \approx 16.7$, so the required concurrency is

Sustaining the rate therefore means keeping roughly 333 requests in flight at once. Because the workload is token-bound, raising the RPM cap would not help; reducing tokens per request — through shorter prompts, smaller completions, or Prompt Caching Savings Calculator — or requesting a higher TPM allocation would.

Variations and application

When requests are request-bound instead, batching several items per call raises effective throughput while trimming tokens does not. Latency enters only through Little's law: doubling $t_L$ doubles the concurrency a fixed rate demands, so faster models relax connection-pool and queue sizing. The concurrency figure feeds client configuration — worker counts, semaphore limits, connection pools — and the sustainable rate feeds cost projections via LLM Monthly Cost Calculator. Because token counts vary per request and providers may queue or burst, the steady-state figures are a planning baseline to validate against live traffic.