Routing

Routing is decided per application. It answers one question for every request: given the registries this application may use, which one gets this call, and with which model. There are four answers, and the console offers them as one Strategy choice. Start with one registry. Fallback and load balancing are things you add once the application works, not things you design up front.

​

Simple routing

One or more registries, no distribution. With one registry every request goes there. With several, the client’s model decides: TrustGate walks the application’s registries in order and uses the first that serves the named model. A name no registry serves is rejected before it reaches any provider. This is the strategy where the client carries the decision. That is fine for a service that knows exactly what it wants, and wrong for anything you would like to re-route later without touching the client.

​

Smart routing

Smart routing chooses a model before generation from an ordered ladder of two or three models. It selects when a conversation starts, then keeps that model while the conversation is warm. Keeping the same model lets subsequent turns reuse the provider’s prompt cache. Routing thresholds are built in; you choose the models and their order.

​

Choose the models

Each slot names a provider and a specific model. Order them from lower cost to higher capability: Start with Two models when Budget and Most capable cover the application’s workload. Choose Three models when a Balanced model adds useful capability at a lower cost than the top slot. Slots may share a provider with different models, or use different providers. Each provider/model pair must be distinct. Choose models that support the application’s inputs, tools and context sizes. Use representative application tasks to check the ladder’s order: Budget should handle routine work, and each higher slot should add capability for harder work.

​

Conversations and prompt caching

The initial request selects a model according to request complexity. Later turns keep that selection while the conversation is warm. After an idle gap longer than the routing cache lifetime, the next request selects again. Use a stable X-Session-Id for each conversation, with session extraction configured on the gateway. Reuse the ID across turns and retries; give a new conversation a new ID. Send the conversation history through the client’s normal message fields, or continue it through the API’s conversation mechanism. The routing lifetime controls how long a model selection remains active between requests. Provider caching follows the provider’s own rules. Keep the prompt prefix stable and use the appropriate cache fields described in Prompt caching.

​

Prioritize quality over savings

Prioritize quality over savings is off by default for a new configuration. With it off, the selected model stays in place while the conversation is warm. Turn it on when conversations often move from routine work to more demanding requests. A harder new user turn can move the conversation one slot upward, once during the warm period. The upgraded model then stays in place until the next idle expiry; the selection does not move downward while warm. Retries and tool continuations retain the selection. An upgrade changes the model serving the conversation and can require a new provider cache write. Use the toggle when the extra capability matters more than preserving the original cache and its savings.

​

Configure in the console

1. Open Applications, select an application, and open Routing.
2. Under Strategy → Load balancing, choose Smart routing.
3. Choose Two models or Three models.
4. Select a Provider and Model for each slot, in the order shown above.
5. Choose whether to enable Prioritize quality over savings, then save.
6. Open Connect and use its generated snippet with "model": "auto".

When changing provider connections, review the ladder so it continues to contain two or three distinct, available provider/model pairs. After rollout, check Analytics for model usage, cache reads and writes, cost and latency. Compare with the application’s previous traffic mix to assess the effect of the setup. Pair routing with LLM Budget when you also need a spend ceiling.

​

Fallback

An ordered chain of registries. TrustGate forwards to the first; on a failure that matches a trigger, it moves to the next, and never re-tries a registry that already failed within the same request. A budget caps how hard it tries — a maximum number of attempts including the first, and a maximum total latency across all of them. When either runs out the client gets the last error, not a synthesised one. Two things about the chain worth knowing before you build one. The same model name has to exist on each registry you want it to reach, or the fallback stops there. And a chain that spans providers — OpenAI, then Azure OpenAI, then Bedrock — is resilience; a chain that goes from a flagship to a cheaper model is a degrade path, and those two intentions want different triggers. Rate-limit triggers suit the first; policy-rejection triggers usually do not.

​

Load balancing

A pool of members, each a registry and optionally a pinned model. An algorithm picks a member per request: Registries with a health check are skipped while unhealthy and rejoin when they recover. Pair load balancing with fallback if a failed pick should retry another path instead of surfacing the error. Load balancing is for models only. Tool traffic routes to a single server, so an application serving MCP never sees these options.

​

What the client sends

The model field in the request is where routing meets the client, and its form has to match the strategy: With several registries and no load balancing, the client must name a model — there is nothing else to break the tie. The application’s Connect tab and the Playground already emit the right form for the strategy configured, so copy from there rather than remembering the rule.

​

Restricting models

Which models an application may reach is set on the application, per registry: either every model the registry serves, or a filtered subset, plus a default for requests that name none. The filter only offers models the registry’s credentials can actually list, so an Azure entry shows its deployments and an OpenAI-compatible endpoint shows what it advertises. A request for a model outside the filter is rejected during resolution, before any provider is contacted. In a load-balanced pool a member’s own model list overrides the registry’s, and a pinned model on a member decides outright. This restriction is not a policy. It costs no policy evaluation and fails closed at resolution, which is why it is the right place for “this service may only call these two models”.

​

Order of resolution

1. Read model: auto, a short name, a qualified name, or nothing.
2. Narrow to the registries this application may use — or the pool, for load balancing.
3. Apply each registry’s model filter; fill in the default where nothing was named.
4. Hand what survives to the strategy, with fallback on failure where configured.

Every step is recorded on the trace, so a request that landed somewhere unexpected can be read back rather than guessed at.