Start with one registry. Fallback and load balancing are things you add once the
application works, not things you design up front.
Simple routing
One or more registries, no distribution. With one registry every request goes there. With several, the client’smodel decides: TrustGate walks the
application’s registries in order and uses the first that serves the named
model. A name no registry serves is rejected before it reaches any provider.
This is the strategy where the client carries the decision. That is fine for a
service that knows exactly what it wants, and wrong for anything you would like
to re-route later without touching the client.
Fallback
An ordered chain of registries. TrustGate forwards to the first; on a failure that matches a trigger, it moves to the next, and never re-tries a registry that already failed within the same request.
A budget caps how hard it tries — a maximum number of attempts including the
first, and a maximum total latency across all of them. When either runs out the
client gets the last error, not a synthesised one.
Two things about the chain worth knowing before you build one. The same model
name has to exist on each registry you want it to reach, or the fallback stops
there. And a chain that spans providers — OpenAI, then Azure OpenAI, then
Bedrock — is resilience; a chain that goes from a flagship to a cheaper model is
a degrade path, and those two intentions want different triggers. Rate-limit
triggers suit the first; policy-rejection triggers usually do not.
Load balancing
A pool of members, each a registry and optionally a pinned model. An algorithm picks a member per request:
Registries with a health check are skipped while unhealthy and rejoin when they
recover. Pair load balancing with fallback if a failed pick should retry another
path instead of surfacing the error.
Load balancing is for models only. Tool traffic routes to a single server, so
an application serving MCP never sees these options.
Smart routing
The console lists it beside the load-balancing algorithms, but it is a different kind of decision: not which member takes the next request but which model this request deserves. Smart routing classifies each prompt into a complexity band and sends it to the tier configured for that band. The bands are fixed labels — you never set a threshold:
Each tier is a registry and a model. Tiers may share a registry — one OpenAI
connection serving
gpt-4o-mini on Simple and gpt-4o on Medium is the common
shape. You need at least two tiers, and each band can be used once.
A prompt whose complexity falls below every configured tier goes to the
cheapest tier, not to a random pick. That makes the cheapest tier
load-bearing in a way the others are not: if its registry is deleted, smart
routing is dropped and the pool reverts to round robin. Losing a middle or top
tier just removes that rung.
The point of smart routing is cost without a product decision per request. Check
Analytics after turning it on: if the tier mix is not what you expected, the
model on Simple is usually too weak for what you thought was simple.
What the client sends
Themodel field in the request is where routing meets the client, and its form
has to match the strategy:
With several registries and no load balancing, the client must name a model —
there is nothing else to break the tie. The application’s Connect tab and the
Playground already emit the right form for the strategy configured, so copy from
there rather than remembering the rule.
Restricting models
Which models an application may reach is set on the application, per registry: either every model the registry serves, or a filtered subset, plus a default for requests that name none. The filter only offers models the registry’s credentials can actually list, so an Azure entry shows its deployments and an OpenAI-compatible endpoint shows what it advertises. A request for a model outside the filter is rejected during resolution, before any provider is contacted. In a load-balanced pool a member’s own model list overrides the registry’s, and a pinned model on a member decides outright. This restriction is not a policy. It costs no policy evaluation and fails closed at resolution, which is why it is the right place for “this service may only call these two models”.Order of resolution
- Read
model:auto, a short name, a qualified name, or nothing. - Narrow to the registries this application may use — or the pool, for load balancing.
- Apply each registry’s model filter; fill in the default where nothing was named.
- Hand what survives to the strategy, with fallback on failure where configured.