Round-robin is a straightforward load balancing strategy in which incoming requests are distributed sequentially among a pool of backend targets. This ensures an even rotation of requests across available services, helping balance the load and improving utilization.

  • Key Benefit: Simple and effective way to spread traffic evenly.
  • Weighted Support: By assigning different weight values to each target, you can influence how often a target is chosen within the round-robin sequence.
  • Health Checks: If health checks are enabled, any unhealthy target is removed from the rotation until it becomes healthy again, ensuring that only valid targets serve requests.

​
Create an Upstream with Round-Robin Strategy

Below is an example command to create an Upstream using the round-robin algorithm. The sample request includes two targets—one for OpenAI and another for Anthropic—each with its own path, provider, credentials, and optional model configurations.

# Create an upstream with round-robin load balancing
curl -X POST http://localhost:8080/api/v1/gateways/{gateway-id}/upstreams \
  -H "Content-Type: application/json" \
  -d '{
    "name": "ai-providers-upstream",
    "algorithm": "round-robin",
    "targets": [
      {
        "path": "/v1/chat/completions",
        "provider": "openai",
        "weight": 50,
        "priority": 1,
        "default_model": "gpt-4o-mini",
        "models": ["gpt-3.5-turbo", "gpt-4", "gpt-4o-mini"],
        "credentials": {
          "header_name": "Authorization",
          "header_value": "Bearer your-openai-key"
        }
      },
      {
        "path": "/v1/messages",
        "provider": "anthropic",
        "weight": 50,
        "priority": 1,
        "default_model": "claude-3-5-sonnet-20241022",
        "models": ["claude-3-5-sonnet-20241022"],
        "headers": {
          "anthropic-version": "2023-06-01"
        },
        "credentials": {
          "header_name": "x-api-key",
          "header_value": "your-anthropic-key"
        }
      }
    ],
    "health_checks": {
      "passive": true,
      "threshold": 3,
      "interval": 60
    }
  }'