AI spend limits: how to prevent runaway LLM costs before they happen

AI bills don't grow the way other cloud bills do. A loop that retries on error, an agent that decides to "try again with more context," a demo link that gets shared further than you expected — any of them can turn a $30 month into a $3,000 one before the invoice arrives. Provider dashboards show you the damage afterwards; per-app checks scale about as well as per-app guardrails do.

An AI spend limit is a budget ceiling enforced at the gateway, before a request is forwarded and the money is spent. You set a dollar budget over a window — an hour, a day, a week, a month — optionally scoped to a provider or a model, and when the budget is reached the gateway stops forwarding matching requests until the window resets. Every app, every key, every agent behind the gateway is covered at once, and none of them can route around it.

This guide covers how limits are defined, how the gateway counts a dollar, sliding versus fixed windows, scoping rules to providers and models, what your code sees at the cap, the fail-open decision, and the handful of budgets that cover most real deployments.

1. What are AI spend limits?

AI spend limits are budget rules enforced at the gateway, ahead of the provider call. Each rule says: no more than this many dollars, over this window, for these providers or models. When the gateway is about to forward a request, it checks the applicable rules; if any budget is already reached, the request is refused and the provider is never contacted.

Your app ──→ AI Gateway ──→ spend check ──→ Provider / local model
                              │
                              │  under budget → forward, meter the cost
                              └─ at the cap   → HTTP 429, nothing spent

Rules:  $5 / day (everything) · $50 / month (OpenAI) · $10 / week (anthropic/claude-opus-*)

Two things distinguish this from the "usage alerts" a provider console offers. It is preventive — the request that would exceed the budget doesn't happen, rather than being reported later — and it is global: the ceiling applies across every application and API key that uses the gateway, so a new script, a teammate's experiment, or a runaway agent all draw from the same, capped pool.

2. Anatomy of a limit

A rule has four fields, all set under AI → Spend limits → Add rule:

The Spend limits list shows each rule's live spend against its budget, so you can see how close a window is to its ceiling without opening the provider's console.

3. How the gateway counts a dollar

Every request the gateway forwards produces a usage record: the route, the provider and model that answered, prompt and completion tokens, latency — and a cost in USD, computed from the token counts and a per-model price table (input price and output price per million tokens, matched by the longest model-name prefix, so gpt-4o-mini-2024-07-18 prices as gpt-4o-mini). A spend rule's "spend" is simply the sum of those costs inside its window, filtered by its scope.

Three consequences worth knowing:

• It's an estimate, and a good one. The figure comes from actual token counts and list prices; it will track your invoice closely but isn't the invoice.
• Local models cost $0. A model running on your own AI Server has no provider price, so its requests add nothing to any budget. A local-first route therefore spends budget only when it falls back to the cloud — the two features reinforce each other.
• Cached responses cost $0 too. A cache hit never reaches a provider, so it's never metered.

4. Sliding vs. fixed windows

The window technique changes what "$100 per month" means, and the difference matters at the boundary.

Pick sliding for anything meant to catch a runaway loop — a burst at 23:50 shouldn't be forgiven at midnight. Pick fixed when the number maps to a budget someone approved for a calendar period.

5. Scoping: everything, a provider, or a model

A rule with no scope applies to all spend. Scoped rules match on the request's resolved primary target — the provider and model the route picked first. For routed requests, the spend rule evaluates the model the gateway selected, not every fallback target the request might eventually try. If the primary fails and the chain falls over to another provider, that call is still metered against whatever rules cover it — the pre-flight check just didn't consider it. Keep an unscoped rule in place so failover spend is always inside some ceiling. With that understood, scoped rules compose naturally with routing:

Model patterns are written as provider/model and accept *. A rule scoped to a provider you don't use simply never matches.

6. Stacking rules

Rules stack, and a request must pass every rule that applies to it. The first rule whose budget is already reached refuses the request; there's no "most specific wins" — all ceilings hold at once. The pattern that results is a small ladder:

$2   / hour   sliding   everything          ← runaway-loop tripwire
$20  / day    sliding   everything          ← daily ceiling
$300 / month  fixed     everything          ← the number finance approved
$100 / month  fixed     providers: anthropic ← keep the frontier tier in check

The hourly rule is the one that matters in an incident: a retry loop that would burn $20 in ten minutes is stopped at $2, long before the daily or monthly rules would notice.

7. What your code sees at the cap

A request that hits a reached budget is answered by the gateway with 429 Too Many Requests and a JSON body naming the rule, and nothing is sent to the provider:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json

{"error": "spend limit reached ($20 / 1 day)"}

Treat it as a budget signal, not a transient error: don't retry in a tight loop — with a sliding window the earliest it can succeed is when old spend ages out, and with a fixed window it's the next boundary. Back off for minutes, surface a "temporarily unavailable" state, or fall through to a local model if you have one. The 429 is the same status the OpenAI SDKs already treat as retry-with-backoff, so most clients do something sensible by default; the important thing is that they don't hammer.

The spend check runs before guardrails, routing failover, and the provider call, so a refused request also costs no guard-model latency.

8. Fail open, by design

The gateway asks the control plane whether the caller is under budget before forwarding, and caches the answer briefly — a minute after an "allowed", twenty seconds after a "denied" so a raised cap takes effect quickly. Two properties follow:

• If the budget service is briefly unreachable, requests are allowed. A cost control must never take your AI features down; the exposure is bounded by the cache window and the outage length, and everything is still metered, so the limit catches up the moment the check succeeds again.
• Slippage near the cap is bounded by the cache TTL. A burst that arrives within the same minute as the last "allowed" answer can overshoot a little. Spend limits are designed to stop runaway spending quickly — a loop is halted in minutes, not at month end — but they are not accounting-precise to the cent. Set the number with that headroom in mind.

9. Five budgets that cover most deployments

Don't start with one giant monthly budget. Start with layers.

1. The tripwire — $2–5 / hour, sliding, everything. Catches loops and abuse within minutes. Everyone should have one.
2. The daily ceiling — $20–50 / day, sliding. The most a bad day can cost.
3. The approved budget — $N / month, fixed, everything, where N is the number someone signed off on.
4. The expensive-tier cap — $X / week, sliding, models: anthropic/*opus* or the equivalent frontier model, so routing failover into the priciest model can't run unchecked.
5. The experiment sandbox — $10 / month, fixed, providers: <new provider> when you're trialling a vendor and want a hard boundary on the trial.

Start with the first three. Add the model cap once Observability shows one model dominating cost, and the sandbox whenever a new key shows up.

10. Common spend-limit mistakes

• Only a monthly limit. A month-sized ceiling lets a loop spend the whole month in an afternoon. The hourly sliding tripwire is the rule that actually saves money.
• Retrying the 429. Under a fixed window, a hot retry loop just generates 429s until the boundary. Back off in minutes, or fail over to a local model.
• Expecting cent-level precision. The allow decision is cached for up to a minute; budget a little headroom below the real ceiling.
• Scoping to a model name that doesn't match. Patterns are provider/model — gpt-4o alone matches nothing; openai/gpt-4o* does.
• Assuming a limit affects other people's keys. Rules are per account: they cover every key and app in your account, not other accounts on the same network.
• Forgetting local models are free. If a limit trips constantly, the fix may be a local-first route rather than a bigger budget.

Frequently asked questions

What happens when an AI spend limit is reached?

The gateway answers matching requests with HTTP 429 Too Many Requests and a JSON error naming the rule, and the provider is never called. Requests resume when enough spend ages out of a sliding window, or at the next boundary of a fixed one.

Are AI spend limits the same as provider usage alerts?

No. A provider alert reports spending after it happens, for that provider only. A spend limit refuses the request before it is sent, and it covers every application and API key that uses the gateway, across all providers.

Can I limit spending for one model or provider?

Yes. Scope a rule to one or more providers, to model patterns written as provider/model with * wildcards (for example anthropic/*opus*), or both. Leave both empty for an account-wide rule.

Do local AI models count toward spend limits?

No. Models running on your own AI Server have no provider price, so their requests are metered at $0. A local-first route only draws on a budget when it falls back to a cloud model.

Can multiple spend limits apply to the same request?

Yes. Rules stack: a request must be under every rule that applies to it, so an hourly tripwire, a daily ceiling, and a monthly budget all hold at the same time.

How accurate is the spend figure?

Cost is computed from actual token counts and per-model list prices, so it tracks the invoice closely. The allow decision is cached for up to a minute, so a burst can overshoot slightly; set budgets with a little headroom.

11. Spend limits aren't the whole story

A production AI gateway needs four layers:

1. Route — where should the request go? (LLM routing)
2. Protect — is the request safe to send, and the answer safe to show? (guardrails)
3. Control — how much can it cost? (spend limits — this post)
4. Optimize — can we avoid calling a model at all? (caching)

They compound: a local-first route spends nothing when the local model answers, the cache spends nothing on a repeat, guardrails refuse the requests that shouldn't be answered at all, and the spend limit caps whatever is left. The complete AI Gateway guide covers all four.

12. Getting started

1. Sign in at hostanywhere.io → AI → Spend limits → Add rule. Spend limits are included on every plan.
2. Create the tripwire: $2 / hour, sliding, no scope. Then the daily and monthly rules.
3. Send a few requests through the Chat Playground and watch the rule's live spend move; set the tripwire to $0.01 for a moment if you want to see the 429 with your own eyes, then put it back.
4. Check Observability → Usage & cost weekly: cost by provider and model tells you which scoped cap to add next.

Full reference in the spend-limits documentation; the walkthrough video shows the rule being created.