Skip to main content
Hybrid high availability uses active/passive data planes in separate regions and Kubernetes clusters. Every data plane connects outbound to the same NeuralTrust SaaS control plane, independently pulls gateway-scoped security configuration, and keeps its own last successfully synchronized version. Only the active region serves LLM and MCP traffic. If that region becomes unavailable, your traffic manager and promotion automation must fence it, verify a passive region, and switch both traffic endpoints. NeuralTrust does not automatically promote customer clusters. Regional active-passive Hybrid architecture. Global LLM and MCP endpoints route to an active data plane in Region A. A passive data plane in Region B independently pulls gateway-scoped security configuration from the NeuralTrust SaaS control plane and can be promoted by customer-operated automation. Runtime and historical data are not replicated between regions.

​
Availability model

This design separates security continuity from data continuity: The passive region is warm: its TrustGate and TrustGuard workloads are running, health checked, and continuously attempting to synchronize, but it does not receive production traffic. Verify its configuration version before promotion. Regional data planes can hold different last-synchronized versions during a control-plane or network interruption. Compare the candidate region’s configuration version with the expected version before promotion.
This model does not replicate runtime or historical data between regions. After failover, the promoted region enforces its own last synchronized policy definitions. Caches, sessions, counters, and regional raw-data history do not move, so decisions that depend on that runtime state can differ after promotion.

​
Regional topology

Deploy each region as an independent Hybrid data plane:
  • A Kubernetes cluster distributed across multiple nodes or availability zones
  • Multiple TrustGate replicas for the LLM/proxy and MCP entry points
  • Multiple TrustGuard replicas on the internal request path
  • Regional PostgreSQL and Redis services
  • Persistent storage for the encrypted last-known-good configuration
  • A DataAgent deployment that is enabled only while the region is active
All regions use the same logical gateway scope and credentials valid for that scope. They independently initiate outbound configuration pulls from NeuralTrust SaaS. Use separate regional load balancers for the LLM/proxy and MCP Services. Place a customer-operated global DNS or traffic-management layer in front of them:
Configure the two global URLs in Settings → Agent Gateway → General. The console continues to store one LLM URL and one MCP URL; the traffic manager controls which regional data plane receives traffic. Keep both URLs on the same selected region. If you use DNS for promotion, account for resolver and client caching that can temporarily delay convergence.

​
Control-plane failure

TrustGate and TrustGuard keep a last-known-good copy of configuration. If the SaaS control plane or the connection to it becomes unavailable:
  1. The active data plane continues processing requests.
  2. TrustGate and TrustGuard continue enforcing the last synchronized security policies.
  3. Passive regions remain ready with their own last synchronized copies.
  4. Configuration changes made during the interruption take effect after synchronization resumes.
Store the encrypted last-known-good file on persistent storage in every region. Running pods can continue from their in-memory configuration, but persistent storage is required when pods restart while the control plane is unavailable. Protect the corresponding CONFIG_SYNC_LKG_KEY through a Kubernetes Secret or approved secret manager.
Do not make configuration changes directly in an individual data plane during a control-plane outage. The SaaS control plane remains the source of truth.

​
Active-region failure

Your global traffic manager and failover automation are responsible for promotion:
1

Detect the failure

Use regional health checks backed by Kubernetes readiness. Treat the region as unavailable when either the LLM/proxy or MCP request path cannot meet your health criteria.
2

Fence the former active region

Remove it from both global traffic endpoints and prevent it from running the active DataAgent. Fencing avoids split brain during network partitions or partial recovery.
3

Promote a passive region

Confirm that TrustGate and TrustGuard are ready and have loaded a synchronized or last-known-good configuration.
4

Enable active-only services

Enable or start DataAgent in the promoted region. Only one DataAgent may be active at any time.
5

Switch both traffic endpoints

Route the global LLM/proxy and MCP URLs to the promoted region. Keep both protocols on the same active region.
6

Verify security enforcement

Run representative allowed and blocked requests, then confirm policy decisions and metadata export before declaring the failover complete.
Requests already in flight in the failed region can fail. Clients should use bounded retries appropriate for their LLM or MCP operation.

​
No cross-region data replication

Each regional data plane uses its own PostgreSQL and Redis services. The failover process does not copy:
  • Raw prompts or responses
  • Redis sessions and counters
  • Rate-limit state
  • Semantic-cache entries
  • DataAgent raw-data history
This trade-off limits the state shared between regions and keeps raw data inside the region where it was processed. The promoted region applies its own synchronized or last-known-good TrustGate and TrustGuard policy definitions to new requests. Decisions that depend on rate-limit counters, sessions, caches, or other non-replicated state can differ after failover. Metadata already exported successfully to NeuralTrust SaaS remains available there. Raw data in the former active region becomes available again only when that region and its local storage recover.

​
Split-brain prevention

Configuration synchronization does not select the active region. Use one external authority—such as your global traffic manager, deployment automation, or a lease service—to control promotion. A promotion must:
  • Remove the former active region from both global endpoints
  • Prevent its DataAgent from reconnecting as active
  • Select exactly one replacement region
  • Record the active-region decision for operators and automation
During an ambiguous network partition, prefer keeping a region passive rather than allowing two active regions.

​
Production checklist

  • At least two regional Kubernetes clusters are deployed.
  • TrustGate and TrustGuard have redundant replicas within each region.
  • Every region independently pulls configuration for the same gateway scope.
  • Last-known-good configuration uses persistent storage.
  • The LKG encryption key and deployment credentials are available in every region.
  • Global LLM and MCP URLs route to only one active region.
  • Regional health checks cover both request paths.
  • Promotion automation fences the previous active region.
  • Only the active region runs DataAgent.
  • Operators accept that runtime and historical data are not replicated.

​
Test the failover

Validate the design before production and repeat the exercise after material infrastructure changes:
  1. Block access from the active region to the SaaS control plane and verify that allowed and blocked requests continue using the last synchronized configuration.
  2. Restart a data-plane pod during the interruption and verify that it restores its persistent last-known-good configuration.
  3. Remove the active region from service and promote a passive region.
  4. Verify the global LLM and MCP URLs separately.
  5. Confirm that the former active region is fenced and that only one DataAgent is connected.
  6. Restore control-plane connectivity and verify the current configuration version in every region.
Your recovery time depends on health-check intervals, traffic-manager convergence, client DNS behavior, and promotion automation. Define and test these values according to your own availability requirements.

Deployment models

Review the responsibility split between SaaS and Hybrid.

Configuration

Configure Kubernetes, regional stores, and ingress.

Secrets

Protect configuration-sync and LKG credentials.

Kubernetes

Review the Kubernetes deployment requirements.