Configure the policy
Open Policies → Library → Rate Limiter.Limit
The window is rolling, not a clock-aligned bucket: the counter is the number
of requests recorded in the last window at any instant, kept at one-second
resolution. There is no reset on the hour, and no moment at which the whole
allowance comes back at once.
The form shows the pair back as a sentence — Allow 60 requests per minute —
which is worth reading before you save. Max requests exactly: with a limit of
60, the sixtieth request in the window passes and the sixty-first is refused.
When limit is exceeded
Group by header never partitions MCP traffic. An MCP tool call reaches the
policy without any HTTP headers attached, so the value is always empty and every
MCP request lands on the shared counter. Configure it if you like; on MCP it does
nothing.
Scope
Mode
This is the only policy type that offers all three modes.How the decision is made
1
The counter for this request is read
The key is the policy, its scope subject (the gateway, or the application), and
the Group by header value if there is one. The value read is how many
requests that key has recorded in the last window.
2
At or above the limit, the mode decides
In Enforce, the request is refused before it reaches the model or the tool.
The caller gets
429, and the request is not counted — a refused request does
not deepen the hole it is in.In Throttle, the request waits and then continues to the next step.In Observe, the violation is recorded and the request continues to the next
step.3
The request is counted and forwarded
The counter is incremented, the window’s expired entries are trimmed, and the
request goes upstream. The response carries the allowance headers, with
Remaining already reflecting this request — a client reading
1 remaining
can spend it without being refused.What the caller receives
A refusal on LLM traffic is429 with these headers, where <scope> is
global for a gateway-wide policy and consumer for a targeted one:
and a refusal that says the rate limit was exceeded, whose limit it was, what
the limit and the window are, and how long to wait before trying again.
On MCP traffic the same refusal comes back as an error carried inside a
200 response — a non-2xx would drop the client’s MCP session — under the code
-32004, with the same headers and the same detail. That code is
deliberately not the generic policy-denied one: it tells a client this is a
matter of timing, and that retrying later will work.
Modes and failures
Limits
- On MCP it counts tool calls and nothing else. Listing the available tools, opening the session, and every other kind of MCP message pass without touching the counter. A client that only lists tools is invisible to this policy.
- Group by header does not work on MCP at all. See the note above.
- Reading the counter and incrementing it are two steps, not one. A burst of genuinely simultaneous requests can all read the same under-limit value and all be admitted, so a limit can be overshot slightly under concurrency. It is a ceiling for sustained traffic, not a mutual exclusion lock.
- Throttle never refuses, and its delay is capped at two seconds.
Retry-Afteris one fixed number, not the time until the next slot actually frees. It is the window unless you changed it, and nothing recomputes it per request.- Every request costs one, whatever it asks the model to do.
- The refusal body is the gateway’s own shape, not your provider’s error shape, whichever wire format your client speaks. A client that only knows how to parse OpenAI or Anthropic errors will see an unfamiliar object.
- The counter lives in the gateway’s shared store. It is not durable accounting: if that store is cleared, every allowance starts again from zero.
Verify the policy
Set the limit low — 2 or 3 — while you test, then raise it. Send these in this order.1
Control — the path works at all
Open the Playground, pick the application the policy applies to, and send:Expect a normal reply. If this fails, the problem is the application, the model
or the provider — not the policy. In Enforce, a
502 here means the gateway
could not reach its counter store at all. Fix it before going on.2
The allowance is being counted
Send the same prompt a few more times, then open the policy and check
Coverage. It tells you how many applications this policy actually reaches —
if the traffic you are sending is not one of them, nothing is being counted and
the next step will never refuse.The running count itself is not something the Playground shows you. The gateway
puts it in the response headers listed above, so a client of your own watches the
remaining allowance fall by one per request; the Playground does not display
response headers.
3
The limit refuses
Keep sending until you have made one more request than Max requests.Expect
429, and a wait matching what Retry after says. Then confirm the
refused request never reached the model: open Activity, click the request,
open its Policies tab, and read Aggregated Latency. The Provider row
should be empty. That the provider leg is empty is the evidence that matters.In Observe you will see no 429 — look for the refusal the policy would
have made on that same Policies tab instead. In Throttle you will see no
429 either; you will see the request take visibly longer and then succeed.4
The partition holds (only if you set Group by header)
Send a request with a different value in the header you named. It must be
allowed, on its own fresh allowance, while the exhausted value is still being
refused.Then send one with the header omitted. It is not exempt — it joins the
shared counter, which is the point of the warning above.
Troubleshooting
Related
- LLM Budget — the same shape of control, for spend rather than volume
- Per-Tool Rate Limiter — a ceiling on tool executions rather than requests
- Policies overview — scope, modes and policy chains
- End-user attribution — why the header you group by is not the same thing as the user on the trace
- Event schema — the fields each decision records