Load Balancing
Overview
Overview: Load Balancing Strategies in TrustGate
Efficient request distribution is critical for performance, reliability, and scalability in modern gateway architectures. TrustGate supports multiple load balancing algorithms that help ensure fair traffic distribution across backend targets, while improving resource utilization and minimizing latency. This overview introduces the four main strategies supported by TrustGate:- Round-Robin
- Weighted Round-Robin
- Least Connections
- Random Selection
Why Load Balancing Matters
Load balancing helps:- Distribute traffic evenly across multiple services
- Prevent overload on any single backend
- Improve fault tolerance by using health checks to bypass unhealthy nodes
- Scale horizontally to handle increasing request volume
Load Balancing Algorithms
1. Round-Robin
Distributes requests sequentially across all available targets.| Feature | Description |
|---|---|
| Behavior | Requests rotate in order (Target A → B → C → A…) |
| Best For | Uniform backends with similar capacity |
| Supports Weights | ✔ Yes |
| Example Use Case | Balanced LLM provider pool with similar models |
2. Weighted Round-Robin
Enhances round-robin by assigning weights to each target, controlling the proportion of traffic received.| Feature | Description |
|---|---|
| Behavior | Targets with higher weight receive more requests |
| Best For | Backends with unequal capacity or cost |
| Customization | ✔ Fine-grained traffic control |
| Example Use Case | 60% traffic to OpenAI, 40% to Anthropic |
3. Least Connections
Routes traffic to the target with the fewest active connections at the time.| Feature | Description |
|---|---|
| Behavior | Tracks real-time active connections |
| Best For | Long-lived or uneven request durations |
| Adaptability | ✔ Dynamically shifts based on load |
| Example Use Case | Chat or streaming-based APIs |
4. Random Selection
Selects a backend target at random for each request.| Feature | Description |
|---|---|
| Behavior | Randomized request routing |
| Best For | Simple, stateless services |
| Overhead | ✔ Minimal tracking required |
| Example Use Case | Lightweight APIs where simplicity matters |
Choosing the Right Strategy
| Scenario | Recommended Algorithm |
|---|---|
| Uniform backend capacity | Round-Robin |
| Different backend resources | Weighted Round-Robin |
| Long-lived connections | Least Connections |
| Stateless quick responses | Random |
Health Checks
TrustGate supports passive health checks, which automatically detect failing targets based on response behavior and temporarily remove them from the load balancing pool. This ensures only healthy targets serve traffic, regardless of the selected strategy.Learn more about each strategy: