Skip to main content
The Upstream object defines the targets and related configurations to which the AI Gateway routes requests. It specifies load balancing algorithms, health checks, credential settings, and other properties needed for managing one or more backend services or providers. Below is a sample Upstream object with its fields and a brief explanation of each.

​
Fields Reference

​
Top-Level Fields

  • algorithm (string) Specifies the load balancing algorithm used for distributing requests among multiple targets. Valid values: round-robin, weighted-round-robin, least-conn, semantic.
  • health_checks (object) Defines active or passive health checks for monitoring the availability of upstream targets.
  • name (string) A human-friendly name for this upstream configuration, often used for clarity or logging.
  • tags (array of strings) An optional array of metadata tags. Useful for organizing, filtering, or searching among multiple upstream configurations.
  • targets (array of objects) Lists one or more backend endpoints (targets) that the gateway routes requests to. Each target contains its own configuration and credentials.
  • embedding_config (object, required when algorithm = semantic) Provider, model and credentials used by semantic load balancing. Fields:
    • provider (string)
    • model (string)
    • credentials (object) with header_name and header_value
  • websocket_config (object) Controls WebSocket behavior when routing to upstream targets. Fields:
    • enable_direct_communication (boolean)
    • return_error_details (boolean)
    • ping_period (string duration)
    • pong_wait (string duration)
    • handshake_timeout (string duration)
    • read_buffer_size (number)
    • write_buffer_size (number)
  • proxy (object) Optional upstream-level proxy configuration with host, port, and protocol.

​
Health Checks

The health_checks object configures how the gateway monitors the health of its upstream targets.
  • headers (object) Key-value pairs representing custom headers to send during health check probes.
  • interval (number) The interval (in seconds) between consecutive health check probes.
  • passive (boolean) When set to true, enables passive health checks in addition to active checks. Passive checks rely on real request/response interactions, marking targets unhealthy if errors are detected.
  • path (string) The URL path used by active health checks to test the availability of upstream targets (for example, /health).
  • threshold (number) The number of consecutive failures required to mark a target as unhealthy (or successes required to restore it to a healthy state).

​
Targets

Each object in the targets array represents an individual endpoint or provider. You can have multiple targets to enable load balancing or to support different services under a single upstream definition.
  • host (string) The hostname or IP address of the target server (for example, api.example.com).
  • port (number) The TCP port on which the target is listening.
  • protocol (string) The protocol used to communicate with the target (e.g., http, https, grpc).
  • path (string) The path appended to the base URL when routing requests to this target.
  • provider (string) When set, designates a provider-style target (e.g., openai, anthropic). In this mode do not set host/port. You must include:
    • credentials (required)
    • models (array, at least one)
    • default_model (must be one of models)
// priority not supported by the current entity; use weight for weighted strategies.
  • weight (number) For load balancing methods that support weights, this value affects how many requests are sent to the target relative to others.
  • headers (object) Additional headers to include in requests sent to this specific target.
  • tags (array of strings) Optional array of tags for metadata or organizational purposes.
  • default_model (string) If applicable, specifies a default model (e.g., for AI or ML deployments) used when routing requests to this target.
  • models (array of strings) Lists model identifiers supported by this target (e.g., AI models or deployment versions).
  • stream (boolean) Enable streaming behavior for this target when supported.
  • insecure_ssl (boolean) Allow insecure SSL for this target (use only in trusted environments).

​
Credentials

The credentials object within each target controls how authentication and related parameters are handled.
  • callow_override (boolean) Indicates whether these credentials can be overridden by other configuration scopes.
  • aws_access_key_id (string) AWS access key ID for authentication (if using AWS-based services).
  • aws_secret_access_key (string) AWS secret access key for authentication (if using AWS-based services).
  • azure_client_id (string) Azure client ID for authentication.
  • azure_client_secret (string) Azure client secret key.
  • azure_tenant_id (string) Azure tenant ID for multi-tenant environments.
  • azure_use_managed_identity (boolean) If true, use Azure’s managed identity instead of explicit credentials.
  • gcp_service_account_json (string) A JSON string with Google Cloud service account credentials.
  • gcp_use_service_account (boolean) If true, the gateway uses GCP service accounts for authentication instead of explicit credentials.
  • header_name (string) The HTTP header name that will carry credentials or tokens.
  • header_value (string) The HTTP header value (e.g., a token) to be used.
  • param_location (string) Indicates where an authorization parameter might be placed (e.g., query, body, or path).
  • param_name (string) Name of the parameter used for authentication.
  • param_value (string) Value of the parameter used for authentication.

​
Usage Notes

​
Multiple Targets, One Upstream

You can define multiple targets to distribute traffic among different hosts or providers, all managed under the same upstream configuration.

​
Load Balancing

The algorithm and weight fields define how traffic is distributed across targets.

​
Health Checks

Proper configuration of health_checks ensures that the gateway only routes requests to healthy targets.

​
Credentials Management

Sensitive credentials should be stored securely (e.g., via a vault or environment variables). The fields shown here indicate where the gateway expects credentials, not how they must be stored.

​
Tags

Use tags to organize or categorize upstream objects and targets, especially helpful in large-scale environments.

​
Examples

​
Example: Semantic Load Balancing Upstream

{
  "name": "upstream-semantic",
  "algorithm": "semantic",
  "embedding_config": {
    "provider": "openai",
    "model": "text-embedding-3-large",
    "credentials": {
      "header_name": "Authorization",
      "header_value": "Bearer ${OPENAI_API_KEY}"
    }
  },
  "health_checks": {
    "path": "/health",
    "interval": 10,
    "threshold": 3,
    "passive": true
  },
  "targets": [
    {
      "host": "svc-1.internal.example.com",
      "port": 443,
      "protocol": "https",
      "path": "/v1/chat",
      "weight": 1,
      "description": "General-purpose chat inference"
    },
    {
      "host": "svc-2.internal.example.com",
      "port": 443,
      "protocol": "https",
      "path": "/v1/chat",
      "weight": 2,
      "description": "High-capacity inference node"
    }
  ],
  "tags": ["prod", "semantic-routing"]
}
Notes:
  • semantic requires embedding_config.
  • Each target.description is required for the semantic algorithm.

​
Example: Provider-Based Target with Streaming

{
  "name": "upstream-openai",
  "algorithm": "round-robin",
  "targets": [
    {
      "provider": "openai",
      "models": ["gpt-4o-mini", "gpt-4o"],
      "default_model": "gpt-4o-mini",
      "credentials": {
        "header_name": "Authorization",
        "header_value": "Bearer ${OPENAI_API_KEY}"
      },
      "stream": true,
      "insecure_ssl": false
    }
  ]
}
Notes:
  • Provider targets must NOT set host/port.
  • credentials, models (≥1) and default_model are required.
⌘I