Skip to main content
Applies to LLM. A rate limit caps how often; a budget caps how much. A handful of long-context requests to an expensive model cost more than thousands of short ones, so a rate limit alone leaves spend unbounded. The budget is the ceiling on tokens or dollars an application, or an end user behind it, may consume in a window — the thing that keeps a runaway agent from becoming an invoice. Caps usage over a window, in tokens or in dollars, and decides what happens when the cap is reached. As with the rate limiter, the budget is one counter gateway-wide and one per application when targeted. The budget is checked before the request goes upstream, using an estimate, and the actual usage is charged after the response.

​
Dollars are only as true as the rates

A dollar budget prices tokens the same way Analytics does: your contract pricing where you set it, otherwise list price. Enter the contract before the budget, or a discounted contract stops too early.

​
Downgrade, not reject

Downgrading keeps the application working at its ceiling — gpt-4o-mini instead of gpt-4o — instead of returning errors. The target must be on the same provider. Reject is for spend that must not happen at all.

​
Where to apply it

When the goal is cost, pair a budget with smart routing: routing lowers the bill on every request, the budget puts a hard stop under it.