LLM routing: how to automatically choose the right AI model

Every AI app eventually hits the same wall: one model can't be the right answer for every request. The cheap model is fine for "summarize this ticket" and wrong for "reason through this contract." The frontier model is brilliant and slow. The local model is free and private but doesn't know everything.

LLM routing lets you stop choosing. Send every request to one endpoint and let a policy pick the best model for that request — local-first, by prompt size, by whether there's an image, by customer tier — with automatic failover when the preferred model isn't available.

Dynamic model switching means your application doesn't have to decide which AI model to use. The gateway evaluates each request and can switch between local, cloud, fast, reasoning, or vision models based on rules you define — and it can change those rules without a deploy.

This guide shows how routing works in the HostAnywhere AI Gateway: what a route is, four routing patterns that cover many common real-world cases, the exact JSON each one needs, how to call a route from your code, and how to prove a rule actually fired. Everything below runs on the free tier.

1. What LLM routing is — and why route at all

LLM routing puts one endpoint between your application and every model it might use. The app sends a request to that endpoint; a policy you control decides which model answers it, and which one to try next if that model fails.

                      ┌──→ Local model (your own hardware)
                      │
Your app ──→ AI Gateway ──→ Cloud model (OpenAI, Anthropic, Groq …)
   "model": "smart"   │
                      ├──→ Vision model (requests with images)
                      │
                      └──→ Fallback model (when the first choice fails)

One endpoint → one route → many possible models

Hard-coding a model name into an app instead creates four problems that only get worse as usage grows:

• Cost. Simple requests sent to a frontier model can cost several times what a smaller model would charge for the same answer — and simple requests are usually most of the traffic.
• Latency. Big models have higher time-to-first-token. Users notice it on the trivial requests most of all.
• Resilience. Providers have outages and rate limits. If the model name is in your code, so is the outage.
• Change. Every provider swap, price change, or new model is a code deploy — usually across several services.

The usual fix is a tangle of if len(prompt) > N branches, retry wrappers, and per-provider SDKs inside the app. Routing moves all of that out of the app and into a policy you edit in one place.

2. What a route is

Your code never names a real model. The gateway resolves the route to an ordered chain of real targets — local models on hardware you own (local/<model>) or provider models (<provider>/<model>) — tries them in order, and fails over automatically when one errors or times out.

A route has three parts:

Conditions can read six attributes of the request:

Operators: eq, ne, contains, in, gt, gte, lt, lte.

Behind every target sits a circuit breaker: after 3 consecutive failures a target is skipped for 30 seconds, so a dead provider doesn't add its full timeout to every request — traffic goes straight to the next target until it recovers.

You can build routes three ways in Dashboard → AI → Routes: the visual Flow editor, a Form, or raw JSON. The examples below use JSON because it's the most precise; every one of them can be pasted into the JSON view as-is.

3. Pattern 1 — Local-first with cloud failover

The route most people start with: try the model running on your own hardware first, fall back to a cloud model if it's slow or down.

When to use it: privacy matters · you have local GPU or Apple-silicon capacity · you want to minimise cloud spend · the local model is good enough for most requests.

{
  "name": "smart",
  "strategy": "fallback",
  "targets": [
    { "type": "local", "model": "local/deepseek-r1:8b", "timeout_ms": 8000 },
    { "type": "cloud", "provider": "openai", "model": "openai/gpt-4o-mini",
      "retries": { "max": 2, "backoff": "exponential" } }
  ]
}

Two settings do the real work. The 8-second timeout on the local target is the promise you're making to users: if the home box can't answer in that time, the request moves on. The retries with exponential backoff on the cloud target absorb a rate-limit blip without failing the request.

4. Pattern 2 — Route by prompt size

The classic cost optimization. Short prompts are often good candidates for a smaller, faster model — a classification, an extraction, a one-line question — while requests carrying a lot of context tend to benefit from a model with a larger context window or stronger reasoning. Length is a useful signal, not a proxy for difficulty: a five-word prompt can be hard and a long one trivial, so treat the threshold as a cost lever you tune, not a rule about intelligence.

When to use it: most of your traffic is short and repetitive · a big share of cost comes from a frontier model answering easy questions · you can measure quality on the cheaper model before committing.

This is a route exported straight from the dashboard (the one in the screenshot below). By default it answers with gpt-4.1-mini and falls back to Claude Opus; when a prompt is 10,000 tokens or more, the rule sends it to Claude Opus directly:

{
  "name": "Demo-11",
  "default": true,
  "strategy": "fallback",
  "targets": [
    {
      "type": "cloud",
      "provider": "openai",
      "model": "gpt-4.1-mini-2025-04-14",
      "timeout_ms": 30000,
      "retries": {
        "max": 2,
        "backoff": "exponential",
        "delay_ms": 0
      }
    },
    {
      "type": "cloud",
      "provider": "anthropic",
      "model": "claude-opus-4-8",
      "timeout_ms": 30000,
      "retries": {
        "max": 2,
        "backoff": "exponential",
        "delay_ms": 0
      }
    }
  ],
  "rules": [
    {
      "when": [
        {
          "attr": "prompt_tokens",
          "op": "gte",
          "value": "10000"
        }
      ],
      "targets": [
        {
          "type": "cloud",
          "provider": "anthropic",
          "model": "claude-opus-4-8",
          "timeout_ms": 30000,
          "retries": {
            "max": 2
          }
        }
      ]
    }
  ]
}

The important idea isn't prompt length itself — it's that routing decisions are made on request attributes rather than on a model name hard-coded in the app. Prompt size is one of six attributes; image presence, headers, metadata, the end user, and the requested model name work exactly the same way.

A few things worth noticing in the export:

• "default": true marks this as the account's default route — requests that ask for the built-in auto model land here.
• Every target carries its own timeout_ms and retries; the rule's target does too. A rule's targets are a full chain of their own — failover still applies inside a matched rule.
• "op": "gte" with "value": "10000" — numeric operators compare numerically when both sides are numbers, so quoting the value is fine.

Pick the threshold from your own traffic: the Usage & cost tab in Observability lists every request with its route and prompt tokens, so you can see where the "simple" requests actually end. 10,000 tokens is a "long document" threshold; for chat-style traffic something like 800 is more typical.

5. Pattern 3 — Send images to a vision model

Text-only models reject image content, and vision-capable models cost more. With has_image the app never has to know which is which:

When to use it: one endpoint serves both text and multimodal requests · your default model is text-only or local · you only want to pay vision prices when there's actually an image.

{
  "name": "assistant",
  "strategy": "fallback",
  "targets": [
    { "type": "local", "model": "local/llama3.2:3b", "timeout_ms": 6000 },
    { "type": "cloud", "provider": "openai", "model": "openai/gpt-4o-mini" }
  ],
  "rules": [
    {
      "when": [ { "attr": "has_image", "op": "eq", "value": "true" } ],
      "targets": [
        { "type": "cloud", "provider": "openai", "model": "openai/gpt-4o" }
      ]
    }
  ]
}

Text requests go local-first; anything with an image jumps to gpt-4o. One route name for the whole assistant.

6. Pattern 4 — Route by tenant, tier, or environment

Sometimes the right model depends on who is asking, not what. Premium customers get the best model; free-tier traffic gets the economical one; staging never touches the expensive provider. Conditions on a header, a metadata key, or the user field handle all three.

When to use it: you sell tiers · several environments share one gateway · specific users or bots should never reach paid models.

{
  "name": "chat",
  "strategy": "fallback",
  "targets": [
    { "type": "cloud", "provider": "groq", "model": "groq/llama-3.3-70b-versatile" }
  ],
  "rules": [
    {
      "when": [ { "attr": "header", "key": "x-plan", "op": "eq", "value": "premium" } ],
      "targets": [ { "type": "cloud", "provider": "anthropic", "model": "anthropic/claude-sonnet-4-5" } ]
    },
    {
      "when": [ { "attr": "metadata", "key": "env", "op": "eq", "value": "staging" } ],
      "targets": [ { "type": "local", "model": "local/llama3.2:3b" } ]
    },
    {
      "when": [ { "attr": "end_user", "op": "in", "value": "qa-bot,load-test" } ],
      "targets": [ { "type": "local", "model": "local/llama3.2:3b" } ]
    }
  ]
}

The app sends x-plan: premium as a header, or {"metadata": {"env": "staging"}} in the body, or "user": "qa-bot" — all standard OpenAI-API fields — and the gateway does the rest. Conditions inside one rule are ANDed, so "premium and large prompt" is just two conditions in the same when.

What routing gives you

A real application uses all four

Picture a customer-support assistant. It makes one API call to one route, and behind that route the four patterns above are just rules:

                 Support request  →  "model": "support"
                                │
                        AI Gateway (route: support)
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
   plain text            has_image = true      x-plan = premium
        │                       │                       │
   Local model  ──fail──▶  Vision model            Claude Sonnet
        │                       │                       │
        └───────── fallback: cloud model (openai/gpt-4o-mini) ─────────┘

Text goes local-first with a cloud fallback; anything with a screenshot goes to a vision model; premium customers get the strongest model; and if any first choice fails, the chain continues. The application code never changes when any of that does.

7. Calling a route from your code

A route is called exactly like a model: the OpenAI-compatible endpoint, your gateway key, and the route name in model. Your app authenticates with a HostAnywhere gateway key (ha-gw-…) — provider keys stay on the gateway and never appear in your code.

curl https://api.hostanywhere.ai/v1/chat/completions \
  -H "Authorization: Bearer ha-gw-..." \
  -H "Content-Type: application/json" \
  -H "x-plan: premium" \
  -d '{
    "model": "chat",
    "messages": [{"role": "user", "content": "Summarize the attached contract in five bullets."}]
  }'

With the OpenAI Python SDK, only two lines change — the base URL and the key:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.hostanywhere.ai/v1",
    api_key="ha-gw-...",          # HostAnywhere gateway key, not a provider key
)

resp = client.chat.completions.create(
    model="smart",                 # the route name — create smart, call smart, inspect resp.model
    messages=[{"role": "user", "content": "Classify this ticket: 'card declined twice'"}],
)
print(resp.model)                  # which real model answered — see below
print(resp.choices[0].message.content)

Running the gateway on your own device instead of the hosted one? Same code — point base_url at the device's gateway address (https://<mesh-ip>:36140/v1) shown under API keys. Routes, keys, and rules are account-wide and apply on every gateway you run.

8. Proving a rule fired

Don't rely on the model's self-description ("I'm ChatGPT…") to know where a request went — models are unreliable narrators of their own identity. Two dependable signals:

1. The response's model field reports the resolved provider model — openai/gpt-4o-mini or anthropic/claude-sonnet-4-5 — not the route name. Print it next to each test request.
2. Observability → Usage & cost lists every request with its route, the provider and model that answered, tokens, cost, and latency. Send a short and a long prompt to a route with a prompt-size rule and you'll see the two rows land on different models.

The Chat Playground (under Tools) does the same interactively: pick the route, send a prompt, and read which target answered.

9. Common routing mistakes

• Call the route by name. Rules and failover live on the route. If your app sends "model": "openai/gpt-4o" directly, it bypasses the route entirely — no rules, no failover.
• auto is the default route. Mark one route as the default and requests for auto resolve to it — useful for apps that don't want to know route names at all.
• Timeouts are per target and add up. A chain of three targets with 30-second timeouts can take 90 seconds to fail. Give local targets short timeouts; let the circuit breaker skip the ones that are down.
• A rule replaces the chain, it doesn't prepend to it. If you want the default targets as a safety net inside a rule, list them again at the end of the rule's targets.
• Provider model names are the provider's. openai/gpt-oss-120b on the Groq provider is OpenAI's open-weight model served by Groq — the prefix names the model family, the provider field names who's serving it.

10. Routing isn't the whole story

A production AI gateway needs four layers:

1. Route — where should the request go?
2. Protect — is the request safe to send, and the answer safe to show? (guardrails)
3. Control — how much can it cost? (spend limits)
4. Optimize — can we avoid calling a model at all? (caching)

HostAnywhere combines all four in one gateway:

• Guardrails inspect every request and response — content safety, prompt-injection detection, PII redaction, custom block lists — and can be set per route, so a customer-facing route can be strict while an internal one is relaxed.
• Caching returns a stored answer when the same request repeats inside a time window, without calling any provider at all.
• Spend limits cap spend over a rolling window, optionally scoped to specific providers or models; at the cap the gateway refuses the request with 429 instead of running up the bill.

All three apply to every route automatically. The complete AI Gateway guide covers each in depth.

11. Getting started

1. Sign in at hostanywhere.io → AI. The Overview walks you through enabling a gateway, adding a provider (or switching on an AI Server on a machine you own), and creating an ha-gw-… key.
2. Open Routes → New route, paste one of the JSON examples above, and save.
3. Point your app's base_url at the gateway, set model to the route name, and send two requests that should land on different targets. Check resp.model.

Routing, failover, guardrails, and spend limits are on the free plan. Full syntax lives in the routes documentation, and the 17-minute walkthrough video shows the whole gateway set up end to end.