Skip to main content
Applies to LLM. A rate limit caps how often; a budget caps how much. A handful of long-context requests to an expensive model cost more than thousands of short ones, so a rate limit alone leaves spend unbounded. The budget is the ceiling on tokens or dollars an application, or an end user behind it, may consume in a window — the thing that keeps a runaway agent from becoming an invoice.
Nothing about the incoming request is estimated or reserved. The budget compares the counter of what has already been charged against the ceiling, and usage is charged only once the answer is known — after the response has been sent. Three consequences, and all three matter:
  • The request that crosses the ceiling is served in full. It is charged afterwards. The first refusal is the request after it.
  • Concurrent requests are all admitted. Ten simultaneous requests against an exhausted-but-not-yet-charged counter all read the same under-budget value.
  • Accrual is asynchronous. Even a sequential client can get one or two more requests through between the crossing request and the first refusal.
A budget of 100,000 tokens does not guarantee you spend 100,000 tokens. It guarantees you stop soon after 100,000, by whatever a request in flight costs. Set it where the overshoot is affordable, and do not treat it as a hard cap on an expensive model. It is a brake, not a wall.
LLM Budget is what the Library calls it. Elsewhere — in what a decision records, for instance — the same type goes by its older name, token_rate_limiter. One policy type, two names; nothing behaves differently.

​
Configure the policy

Open Policies → Library → LLM Budget.

​
Budget

The form shows the pair back as a sentence — Allow 100000 tokens per hour.
The console offers Seconds, and the gateway does not honour anything under a minute. Any window shorter than 60 seconds is silently raised to 60 seconds. The summary line will still read per 30 seconds. Choose a window of a minute or more so the form and the gateway agree.
The window is fixed, and it opens on the first charge. It is not a rolling window and not aligned to the clock: the counter is created when usage is first charged to it, lives exactly the window’s length, then disappears and the whole allowance returns at once. An hour-long budget exhausted at 10:59 is empty for one minute and full again at 11:00 — if the counter happened to open at 10:00.

​
When limit is exceeded

Downgrading keeps the application working at its ceiling — gpt-4o-mini instead of gpt-4o — instead of returning errors. Reject is for spend that must not happen at all.
A downgrade that cannot be applied refuses instead — but a wrong target is not always caught. The target must be a single model name. A pool alias, a wildcard, a target that names a provider other than the one serving the request, or — where the application restricts which models it may call — a model outside that list is not valid, and a budget configured to downgrade to one behaves exactly like Reject request, with nothing in the policy to say so.What is not checked is whether the provider serving the request actually has the model you typed. A plain model name is taken at its word: a typo, or a model that belongs to another provider written without naming that provider, is applied as a downgrade rather than refused. The request goes out under the new name, the downgrade header is set, and the provider answers with its own “no such model” error — so what you get is a provider failure, not the clean refusal you might have expected. Copy the target from the model’s entry in Registry rather than typing it, and exercise it once against real traffic before you rely on it.When the downgrade does apply, the response carries X-NeuralTrust-Model-Downgraded: <original>→<target>. That header is the only signal the caller gets; the status is a plain 200.

​
Per model limits

Optional. Each line is a Model pattern — a model name, or a wildcard such as claude-opus-* — a Max, and its own Time window. A request is checked against the overall budget and against the most specific per-model rule matching its model. Exceeding either triggers the behaviour, and usage is charged to both. An exact slug beats a wildcard; a longer wildcard beats a shorter one.
The form makes the overall budget mandatory, so per-model rules are always a tightening, never the whole policy. A policy that came from somewhere else can carry per-model rules with no overall budget — and then a model matching no rule has no budget at all. If your policies are built that way, one of them should be a catch-all.

​
Advanced Settings

Group by header is a partition, not an exemption. A request that does not carry the named header is not skipped: it falls back to the policy’s own counter, shared with every other request that omits it. It is also a raw header read, not the end-user attribution pipeline, which never reaches policy decisions.

​
Scope

​
Mode

Enforce and Observe. Observe is a real dry run here, and a better one than most policies offer: usage is still charged, the headers are still emitted, and a request over the ceiling is still recorded — it is simply never refused or downgraded. Run a window in Observe, read the counts, then switch.

​
Dollars are only as true as the rates

A dollar budget prices tokens the same way Analytics does: your contract pricing where you set it, otherwise list price. Enter the contract before the budget, or a discounted contract stops too early.
A model with no price accrues nothing against a dollar budget. If neither your contract pricing nor the built-in catalogue knows the model, its usage is costed at zero and the request is free as far as the budget is concerned. A warning is logged and the request is marked unpriced; nothing is refused. A self-hosted or newly released model is the usual case. Price it in Contract pricing, or use a token budget for that traffic.

​
How the decision is made

1

The counters for this request are read

The overall budget’s counter, and the counter of the most specific per-model rule matching the request’s model. Both keyed by the policy, its scope subject, and the Group by header value if there is one.Nothing about this request is measured. No prompt is tokenised, no cost is estimated, nothing is reserved.
2

At or above either ceiling, the behaviour applies

In Enforce, Reject request refuses with 429; Downgrade model rewrites the request’s model and lets it continue. In Observe, the violation is recorded and the request continues unchanged.The ceiling that is reported — in the headers, and on the request’s Policies tab — is the first one found to be breached.
3

The request is forwarded

Under budget, it goes upstream untouched, and the response carries the remaining allowance.
4

After the response, the actual usage is charged

The gateway reads the usage the provider reported, applies Counting, converts to dollars if the unit is dollars, and adds it to every window in play. This happens after the response has been sent to the caller, on its own, so it never delays the answer — and never gates it either.

​
What the caller receives

Every response, refused or not, carries the state of the reported window: So the unit, the scope and the window are on every refusal, whichever unit the budget is in — a token refusal is not a lesser one. Both reset headers — X-Ratelimit-Reset-Tokens and X-Budget-Reset — are a number of seconds with a seconds suffix: 3600s, not 3600. A client that reads them as a bare number will not parse them. A refusal is 429, and the response says which budget was exhausted — the token one or the dollar one — whether the ceiling was the gateway-wide one or the one targeted at this application, and which window was breached.

​
Modes and failures

In Observe, a counter-store failure still refuses the request. The mode is read after the counter, so a store that cannot be reached fails the request before anything has consulted the mode. A policy you attached only to watch can take an application down with 502s.
A stream the client abandons costs you money the budget never sees. Usage is charged only when the gateway has the whole response. A client that disconnects mid-stream — or a streamed answer larger than the gateway’s post-response buffer — leaves the tokens spent at the provider and the counter untouched. An agent that cancels aggressively can run well past its budget without ever tripping it.

​
Limits

  • No pre-flight estimate, no reservation. See the warning at the top of this page. The ceiling is enforced against what has already been charged.
  • Accrual is asynchronous, so the counter lags the traffic by roughly one response.
  • The windows are fixed, open on the first charge, and return the whole allowance at once when they close.
  • Windows shorter than a minute are raised to a minute, without the form saying so.
  • LLM traffic only. MCP (Model Context Protocol) tool calls are outside this policy entirely; the Per-Tool Rate Limiter is the tool equivalent.
  • Only usage the provider reports is charged. A response without a usage block is free.
  • Abandoned and oversized streams are not charged.
  • Unpriced models are free against a dollar budget.
  • Prompt-cache reads are free against a token budget. Input tokens the provider served from its prompt cache are subtracted before anything is counted, so a long system prompt reused across a conversation charges the budget once and then stops charging for itself. There is no setting in the console to count them. A dollar budget does count them, at the provider’s cheaper cache-read rate rather than the full input rate. An application that caches heavily will therefore travel much further inside a token budget than its raw token counts suggest.
  • A downgrade target must be a single model name the application may call, and not a pool alias or a wildcard; anything else silently becomes a refusal. A name the request’s provider does not serve is not caught — it is applied, and the provider errors.
  • Retry-After is not sent. The refusal carries the reset in the budget headers instead, so a client that only reads Retry-After learns nothing about when to come back.
  • The refusal body is the gateway’s own shape, not your provider’s error shape, whichever wire format your client speaks.
  • The counters live in the gateway’s shared store. They are operational state, not accounting: if that store is cleared, every budget starts again from zero. Use Analytics for what was actually spent.

​
Verify the policy

Set Max low — a few hundred tokens, or a fraction of a dollar — and the window to a minute while you test. Send these in this order.
1

Control — the path works at all

In the Playground, pick the application the policy applies to, and send:
Expect a normal reply. If this fails, the problem is the application, the model or the provider — not the policy. In Enforce, a 502 here means the gateway could not reach its counter store. Fix it before going on.
2

The budget is being charged

Pick a specific model in the Playground first, not Auto. With Auto the gateway chooses the model for you, and on that path the Playground does not ask the provider to report what the request used. Providers that only report it when asked — OpenAI among them — then report nothing, so nothing is charged and a budget you are watching never moves however many prompts you send. Open the model selector and choose one model by name.Then send the same prompt a few more times and look those requests up in Activity. Open one and read its Policies tab: this policy should be listed with the usage it charged.The remaining allowance is not something the Playground shows you. The gateway puts it in the response headers listed above, so a client of your own watches it fall as it goes; the Playground does not display response headers.If nothing is being charged, work down this list:
  • Auto is still selected in the Playground’s model selector. Pick a model by name and send again.
  • The traffic is not in this policy’s scope. Check Coverage on the policy, and that its Scope does not read Matches nothing.
  • The responses carry no usage at all.
  • On a dollar budget, the model has no price. The request’s Policies tab marks it as unpriced. Price it in Contract pricing before assuming the policy is broken.
3

The ceiling acts

Keep sending. Expect to cross the ceiling without being refused — that request is served in full — and to be refused on a later one. That is the policy working as designed, not a fault.With Reject request, expect 429 and a refusal naming the budget that was exhausted. Then confirm the refused request never reached the model: open Activity, click the request, open its Policies tab and read Aggregated Latency — the Provider row should be empty. That the provider leg is empty is the evidence that matters.With Downgrade model, expect 200 and the downgrade header on the response, naming the model asked for and the one that answered. Read which model answered, not just the status. A 429 instead means the target is not valid for this request; an error from the provider about an unknown model means the target was applied but that provider does not serve it.
4

The partition holds (only if you set Group by header)

Send a request with a different value in the header you named. It must be allowed on its own fresh budget while the exhausted value is still refused. Then send one with the header omitted: it is not exempt — it joins the shared counter.
Every decision this policy takes is recorded, in Observe mode too, where nothing is blocked. To read one, open Activity, click the request, and open its Policies tab: the policies chain lists this policy and the decision it made. The event schema is the reference for everything a decision records.

​
Troubleshooting

  • Rate Limiter — the same shape of control, for volume rather than spend
  • Smart routing — use an ordered model ladder with cache-aware conversation routing; the budget adds a spend ceiling
  • Contract pricing — the rates a dollar budget counts with
  • Policies overview — scope, modes and policy chains
  • Event schema — the fields each decision records