High availability

Availability is a ladder, not a single design. Climb it only as far as the failure you actually have to survive — each rung costs more to operate than the one below it.

This page covers every self-hosted model: Hybrid, External, and Central control plane. The tiers are the same for all three. What changes per model is small and listed at the end.

The ladder

The four tiers plotted on cost and operational complexity against availability. Tier 1, zone resilient, sits lowest on both axes and is boxed and badged 99.95% cloud SLA. Tier 2, cluster resilient, costs more. Tier 3, region resilient, costs more again. Tier 4, always on, is the most expensive. The line through them flattens towards the top right, because each rung costs more and adds less. A footnote records that 99.95% is the monthly uptime GKE, EKS and AKS each commit to for a regional, zone-redundant cluster, that it covers the Kubernetes API server, and that your own figure also depends on your nodes and datastores.

Tier	Survives	Costs you
1. Zone resilient one cluster, across zones	A node, or an entire availability zone	Node pools in three zones; nothing else to run
2. Cluster resilient twin clusters, one region	Losing a cluster, and bad upgrades	A second cluster and a traffic switch
3. Region resilient two regions, active/passive	Losing an entire region	The above, plus DNS promotion and a datastore runbook
4. Always on two regions, active/active	Losing a region, with no promotion of traffic	Each region sized for all the traffic, geo-aware routing, and per-region counters for good

Tier 1 is enough for most deployments. It is also the only rung the public clouds put a number on, because it is exactly what they sell: a regional, zone-redundant Kubernetes cluster. GKE, EKS and AKS each commit to 99.95% monthly uptime for one. Read that as the ceiling your design works within rather than a promise about your traffic, because it covers the Kubernetes API server and not the workloads on it. Tiers 2 to 4 are topologies you build yourself, so no vendor underwrites them. Start at tier 1 and stop there unless a regional requirement says otherwise. The tiers are additive: each one assumes you have already done the ones below it, so tier 3 is tier 1 inside each region plus a promotion procedure, and tier 4 is tier 3 with both regions taking traffic. The chart now ships most of tier 1: the request path defaults to two replicas and spreads itself across zones. What is left to you is the detector replicas, the disruption budgets and the datastores.

Two invariants

Everything below is easier to reason about once these are fixed. They hold at every tier, in every deployment model.

Exactly one writable PostgreSQL primary

A single writer is what prevents split-brain. Two clusters cannot diverge if there is only one place to write, which turns failover into a datastore operation rather than a distributed decision.

Exactly one active DataAgent per gateway scope

Two DataAgents on the same scope register duplicate streams. This is why dataagent.replicas is 1 and must stay 1 — it is a correctness constraint, not a capacity setting.

Tier 1: one cluster, across zones

Spread the node pool over three availability zones and use managed PostgreSQL and Redis with automatic failover inside the region. The chart already runs the request path at two replicas and spreads it across those zones. There is no promotion procedure and no DNS work.

Tier 1: one regional Kubernetes cluster whose node pool spans three availability zones. Zone a, zone b and zone c each run agentgateway, trustguard and the firewall. Below the cluster, managed PostgreSQL is multi-AZ with primary and standby in different zones, and managed Redis holds cache and counters that rebuild in seconds. A note records that the chart ships two replicas and spreads them across zones, leaving you the firewall worker replicas, the disruption budgets and the managed datastores.

What the chart does, and what you still set

The chart now ships tier 1 most of the way: the request path defaults to two replicas and spreads itself across zones. Three things are still yours to set — the replica count on the detectors, the disruption budgets, and the datastores, which is the one that decides whether you are actually at tier 1.

Replica counts

The request path defaults to two replicas, so a fresh install already survives losing a node. These are the numbers worth checking before you call a cluster tier 1.

Workload	Chart key	Default
Gateway proxy (LLM)	`agentgateway.dataPlane.replicas`	`2`
Gateway MCP	`agentgateway.mcp.replicas`	`2`
Gateway admin — External only	`agentgateway.controlPlane.replicas`	`2`
TrustGuard data	`trustguard.dataPlane.replicas`	`2`
TrustGuard control — External only	`trustguard.controlPlane.replicas`	`2`
Firewall gateway	`firewall.firewall.gateway.replicas`	`2`
Firewall workers	`firewall.firewall.workerDefaults.replicas`	`1` — raise it, these are the detectors
Data-plane API	`data-plane-api.dataPlane.components.api.replicas`	`2`
Control-plane API	`control-plane-api.controlPlane.components.api.replicaCount`	`2`
Console	`control-plane-app.controlPlane.components.app.replicaCount`	`2`
DataBridge	`databridge.replicas`	`2`
DataAgent	`dataagent.replicas`	`1` — leave at 1, see the invariants

The firewall workers hold the SLM detectors and are the expensive pods, which is why they still ship as singletons. On a GPU node pool, raising them is a cost decision as much as an availability one. autoscaling.enabled is false everywhere. Leave it that way until you have a capacity reason — autoscaling is a throughput decision, not an availability one, and a minimum replica count is what tier 1 actually depends on.

Zone spread, and budgets for draining

Replicas alone do not survive a zone loss: nothing stops Kubernetes putting them both on one node, and nothing stops a drain taking them at the same time. Two separate settings cover those. Spreading is on by default. Every workload renders topologySpreadConstraints across topology.kubernetes.io/zone and then kubernetes.io/hostname. They use whenUnsatisfiable: ScheduleAnyway, which makes them a preference rather than a requirement — so they cannot block scheduling on a single-zone or single-node cluster, and there is nothing to turn on for the common case. Disruption budgets are opt-in, behind one switch. A budget stops a node drain evicting every replica at once:

global:
  highAvailability:
    podDisruptionBudget:
      enabled: true

That covers every component. Resolution is the same everywhere — an explicit per-component value wins, then global.highAvailability, then the built-in default — so one component can opt out again:

agentgateway:
  dataPlane:
    podDisruptionBudget:
      enabled: false        # this one component stays out
      # maxUnavailable: 1   # or shape the budget instead

A budget never renders on a single replica. One replica with a budget pins disruptionsAllowed at 0 and deadlocks every node drain on that node, so the chart suppresses it below two replicas — including for DataAgent, which must stay a singleton.

The same three keys — podDisruptionBudget, topologySpreadConstraints and affinity — are accepted on every component, and global.affinity merges underneath a per-component affinity the way global.nodeSelector does. Then confirm the replicas really did spread:

# which node each gateway replica landed on
kubectl -n neuraltrust get pods \
  -l app.kubernetes.io/name=agentgateway,app.kubernetes.io/component=data-plane \
  -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName --sort-by=.spec.nodeName

# and which zone those nodes are in — two replicas in one zone is not tier 1
kubectl get nodes -o custom-columns=\
NODE:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone

Managed datastores are what make tier 1 real

The chart deploys PostgreSQL and Redis in-cluster by default, so a first install needs nothing provisioned. That default is for a proof of concept. Each store runs as a single pod, directly under the request path, so spreading the gateway over three zones buys you nothing while one Redis pod can still take the whole path down with it.

Tier 1 is not reached until both datastores are managed services with automatic failover inside the region. Set deploy: false and point at instances outside the cluster:

global:
  postgresql:
    deploy: false
    host: <your managed PostgreSQL>
  redis:
    deploy: false
    host: <your managed Redis>

See managed stores for the full values, and datastore sizing floors for the minimum shapes.

Tier 2: twin clusters in one region

Two clusters side by side, both pointed at one PostgreSQL primary and one Redis primary. Because they are in the same region, sharing both stores adds no network hop and loses nothing on a switch: state is identical from either side.

Tier 2: a traffic switch you operate holds one hostname pointed at one cluster at a time. Cluster 1 is serving production traffic and cluster 2 is the standby where you roll the next version first. Both run agentgateway, trustguard and the firewall with replicas across three zones on the same chart version. Because they sit in one region they share one set of stores: one writable PostgreSQL primary and one Redis primary.

What it buys you is an escape from a broken cluster or a bad upgrade: roll the second cluster, move traffic, and keep the first as your way back. Keep both clusters on the same chart version and the same values. A twin cluster that has drifted is not a rollback target — it is a second thing to debug during an incident.

Tier 3: two regions, active/passive

Both clusters are full installs. The passive one is warm: its workloads run, pass health checks, and keep synchronizing, but it receives no production traffic.

Tier 3: two regions in active and passive. A traffic manager you operate holds one hostname per endpoint resolving to the active region, re-pointed on promotion. Region A is active, serves every request and runs the only DataAgent. Region B is passive and warm, Ready but taking no production traffic, with its DataAgent stopped. Each region has its own region-local Redis. Below both, one writable PostgreSQL primary streams to a cross-region read replica. Rate limits are counted per region because Redis is region-local by design.

Both read and write one PostgreSQL primary, which streams to a cross-region read replica promoted only if the primary’s region fell over.

Why the two datastores are treated differently

PostgreSQL — one writable primary, one cross-region read replica. It holds the durable state, so a single writer is what prevents split-brain. Promotion is a datastore operation rather than a distributed decision, and your RPO is the replication lag. Redis — region-local, one per cluster. Redis carries semantic cache, rate-limit counters and evaluation progress, and has no persistence requirement. Reaching across a region for that on every request costs latency on the hot path and buys nothing, because the data is worth seconds. After a promotion the new active cluster starts with cold counters and rebuilds them in seconds. The consequence is that rate limits are counted per region. With one active region at a time that is invisible, except in the switchover window, when a client could briefly get a fresh allowance. With two regions active at once the same behaviour becomes permanent — that is the trade tier 4 asks you to accept.

What each failure looks like

Failure	Effect
A node or a zone	Remaining replicas absorb the traffic; no operator action
The active cluster	Your traffic manager and promotion automation switch the endpoints to the passive cluster
The PostgreSQL primary	Managed failover inside the region; losing the region means promoting the read replica and repointing both clusters
A region’s Redis	That cluster loses cache and counters and rebuilds them; the other region is untouched
The control plane, or your egress to it	Data planes keep enforcing their last synchronized configuration. Changes apply once sync resumes. Hybrid and Central only — External has no such dependency

Restarts during a control-plane outage

Running pods survive a control-plane outage from their in-memory configuration, but a pod that restarts during one needs the last-known-good file on disk. The chart mounts it on an emptyDir, which does not survive a restart, so a restarted pod with no control-plane connectivity will not become Ready. If a cluster has to tolerate restarts mid-outage, move that mount to persistent storage. The CONFIG_SYNC_LKG_KEY that encrypts it is generated per cluster and needs no distribution. See Secrets for the key, and Config sync for the mechanism.

Do not make configuration changes directly in one data plane during a control-plane outage. The control plane remains the source of truth.

Promote the passive cluster

Detect and fence

Use regional health checks backed by Kubernetes readiness on every published endpoint. Remove the failed cluster from all of them and stop its DataAgent, so it cannot come back mid-recovery and register a second stream.

Point PostgreSQL at a writable primary

If the primary is in the failed region, promote the read replica and update global.postgresql.host in the surviving cluster. If the primary is unaffected, there is nothing to do here — this is the case the single-primary design is buying you. Redis needs no attention either way, because the surviving cluster already has its own.

Verify configuration and start active-only services

Confirm the request path is Ready with a current or last-known-good configuration, then enable DataAgent. Only one DataAgent may be active.

Switch every traffic endpoint

Route the published hostnames to the promoted cluster. Keep all protocols on the same cluster, and remember that DNS-based promotion is delayed by resolver and client caching.

Verify enforcement

Run representative allowed and blocked requests, then confirm policy decisions and telemetry before declaring the failover complete.

Requests already in flight in the failed cluster fail. Clients should use bounded retries appropriate for their LLM or MCP operation.

Before you rely on it

Both clusters are deployed, health checked, and on the same chart version.
Every request-path workload has redundant replicas across zones in each cluster, with PodDisruptionBudgets enabled.
PostgreSQL is managed, with one primary and a cross-region read replica.
Each region has its own managed Redis, and you accept per-region counters.
Last-known-good configuration is on persistent storage.
Only the active cluster runs DataAgent.
Every published hostname resolves to exactly one cluster.
You have rehearsed: block egress to the control plane and confirm traffic still flows; restart a pod during that outage; promote the passive cluster and switch every endpoint.

Tier 4: two regions, active/active

Both regions take production traffic at the same time. Nothing is promoted when a region fails: your traffic manager stops sending clients there, and the survivor carries everything. That is the whole gain — and the reason it is the most expensive rung to run honestly.

Tier 4: two regions both active. A traffic manager routes clients to whichever region is nearer, and every hostname resolves to both. Region A hosts the PostgreSQL primary so its writes stay in-region, and runs the only DataAgent. Region B also serves traffic, but every PostgreSQL write it makes crosses the region boundary, and its DataAgent stays stopped. There is still one writable primary streaming to a cross-region read replica. Each region must carry all of the traffic alone, and per-region rate limits become permanent.

Talk to us before you build this. Tier 4 is not a values change. It is a design conversation about your traffic manager, your write latency budget and how your clients are distributed. The platform supports it inside the same two invariants as every other tier, and those invariants are exactly what make it cost more than it looks.

What stays the same

Still one writable PostgreSQL primary. Both regions write to it. The region that does not host it pays a cross-region round trip on every write — raw payloads in Hybrid, and additionally configuration and console state in External. Measure that latency against your budget before committing, and put the primary in the region that writes most. Losing the primary’s region is the same promotion as at tier 3; the difference is that the surviving region is already serving, so only the datastore step and DataAgent remain. Still exactly one DataAgent. It reads the single PostgreSQL primary, so one agent already serves both regions’ data. Run it in the primary’s region, and start the other one only as part of a promotion. Still region-local Redis, and still independent configuration. Each region pulls its own configuration and keeps its own last-known-good copy, exactly as at tier 3.

What it costs you for good

Consequence	What to do about it
Rate limits are counted per region, permanently. A client whose requests land in both regions can get up to twice its allowance.	Route by geography or client identity so that each client always lands in one region. Do not use round-robin. There is no globally exact counter across regions.
The semantic cache is per region. Hit rates fall until each region has warmed its own.	Accept it. Pinning clients to a region also keeps their cache hits in one place.
Each region must carry all of the traffic on its own. The day one region fails, the other takes everything.	Size and load-test each region for the whole peak, not half of it. Two half-sized regions are a tier 3 without the runbook.
Health checks decide who serves. A region that is Ready but slow keeps taking traffic.	Back the traffic manager’s checks with Kubernetes readiness on every published endpoint, and check the request path rather than a static page.

Before you rely on tier 4

Everything on the tier 3 checklist, plus:

Every client is routed to exactly one region under normal conditions, and you know how.
Each region has been load-tested carrying the whole peak alone.
You have measured cross-region write latency from the far region and accepted it.
You have rehearsed withdrawing a region from the traffic manager and watched the other absorb the traffic.

What changes per deployment model

The ladder is identical in all three. These are the only differences, and each model’s page covers its own in detail.

Model	What is specific to it
Hybrid	Both clusters pull the same gateway scope from NeuralTrust SaaS independently and keep their own last-known-good copy. DataAgent runs in the active cluster only.
External	Configuration lives in PostgreSQL, so a second cluster sees the same policies the moment it starts. ClickHouse is in-cluster and does not follow a promotion — at tier 4 each console permanently sees only its own region’s telemetry unless you replicate it. Console sessions live in the region that issued them, so at tier 4 pin the console hostname to one region. Upgrade one cluster at a time — both consoles run migrations against one database.
Central control plane	Two questions, not one: the remote data planes climb this ladder, and the central cluster usually sits on tier 1 because it is not on the request path. If it does go to tier 3, four published hostnames have to follow the promotion, certificates must be valid from both central clusters, and the platform Secret’s signing keys must be identical in both.

Recovery time

Recovery time depends on health-check intervals, traffic-manager convergence, client DNS behavior, and datastore promotion — all of them yours, not the chart’s. Measure yours rather than assuming it, and re-measure after any change to the traffic manager or the datastore topology.

Next steps

Requirements and capacity

Node pools, datastore sizing floors, and the ports between components.

Configuration

Managed stores, ingress, TLS, and every values switch.

Validation

The checks that confirm a cluster is actually serving what you think.

Troubleshooting

What to look at when a replica will not become Ready.

Overview

Organization

Identity & Access

Telemetry

Audit & Compliance

Deployment

The ladder

Two invariants

Exactly one writable PostgreSQL primary

Exactly one active DataAgent per gateway scope

Tier 1: one cluster, across zones

What the chart does, and what you still set

Replica counts

Zone spread, and budgets for draining

Managed datastores are what make tier 1 real

Tier 2: twin clusters in one region

Tier 3: two regions, active/passive

Why the two datastores are treated differently

What each failure looks like

Restarts during a control-plane outage

Promote the passive cluster

Before you rely on it

Tier 4: two regions, active/active

What stays the same

What it costs you for good

Before you rely on tier 4

What changes per deployment model

Recovery time

Next steps

Requirements and capacity

Configuration

Validation

Troubleshooting

​The ladder

​Two invariants

Exactly one writable PostgreSQL primary

Exactly one active DataAgent per gateway scope

​Tier 1: one cluster, across zones

​What the chart does, and what you still set

​Replica counts

​Zone spread, and budgets for draining

​Managed datastores are what make tier 1 real

​Tier 2: twin clusters in one region

​Tier 3: two regions, active/passive

​Why the two datastores are treated differently

​What each failure looks like

​Restarts during a control-plane outage

​Promote the passive cluster

​Before you rely on it

​Tier 4: two regions, active/active

​What stays the same

​What it costs you for good

​Before you rely on tier 4

​What changes per deployment model

​Recovery time

​Next steps

Requirements and capacity

Configuration

Validation

Troubleshooting

The ladder

Two invariants

Tier 1: one cluster, across zones

What the chart does, and what you still set

Replica counts

Zone spread, and budgets for draining

Managed datastores are what make tier 1 real

Tier 2: twin clusters in one region

Tier 3: two regions, active/passive

Why the two datastores are treated differently

What each failure looks like

Restarts during a control-plane outage

Promote the passive cluster

Before you rely on it

Tier 4: two regions, active/active

What stays the same

What it costs you for good

Before you rely on tier 4

What changes per deployment model

Recovery time

Next steps