Architecture
How the planes connect
Config sync and DataBridge are outbound only. Your cluster dials them;
nothing at NeuralTrust initiates those, so you open egress and never ingress
for them.
The console is the one exception. The Dataplane URL you register there is a
URL NeuralTrust calls, so your published LLM and MCP entry points must accept
inbound HTTPS from a single NeuralTrust source IP. Agree that rule with your
network team early — it is the requirement security reviews most often push
back on. The address is in Network.
TrustGate and TrustGuard never talk to SaaS directly for metadata. They emit
OTLP to a collector co-located with DataAgent inside your cluster, and that
collector is what egresses.
What runs
Products are selected with
global.products.trustgate,
global.products.trustguard, and global.products.dataPlane. All default to
off and at least one must be true.
DataAgent is worth understanding because it is unusual: it exposes no inbound
service. It dials out to DataBridge and holds the stream open, and queries
arrive over that stream. There is nothing to expose and nothing to allow
inbound. It also runs single-replica by design — two would register duplicate
streams. Each enabled product needs its own enrolled DataAgent; only a
red-teaming-only install (global.products.dataPlane alone) runs without one.
TrustGate admin, the console, and analytics are not deployed in Hybrid —
they run on SaaS. There is no in-cluster ClickHouse.
Prerequisites
- A Kubernetes cluster with an ingress controller, and Helm 3.8+
- Roughly 3–4 workers at 8 vCPU / 16–32 GiB — a starting point, see Capacity
- A base domain you can point at the cluster ingress, for example
platform.example.com - The network rules in Network below
- A reachable PostgreSQL and Redis
- The registry key NeuralTrust sends you when your account is provisioned — Container images
Check your datastores first
The two most common first-install failures are a datastore that is not actually reachable and one that is reachable but silently wrong. Redis is required. Both gateways validateREDIS_HOST at boot and refuse to
start without it, even though a Hybrid data plane does not use PostgreSQL for
its own configuration. TrustGate uses Redis for rate limiting and semantic
caching on the request path.
PostgreSQL is where your raw payloads live. Prompts and responses are
written there and never leave your cluster.
The chart deploys both in-cluster by default so a proof of concept can start
without provisioning anything. Production should set deploy: false and point
at managed instances outside the cluster — see
Managed stores.
An unreachable store fails loudly. The pattern to watch for is a component
whose default quietly points at localhost: it comes up healthy and looks fine
until you use the feature that needs it. Set every host explicitly.
Network
Allow TCP 443 from your cluster egress — and from any NAT or proxy in front of it — to these hosts. Prefer hostname rules where your firewall supports DNS-based allowlists; the IPs are for static ACLs. If DNS resolves differently in your region, trust DNS and ask NeuralTrust support to refresh the list.
Allow the telemetry host matching the region your tenant is provisioned in.
Product metadata OTLP does not leave the pods directly — the co-located egress
collector forwards to the ingest edge for your region, so that is the host to
list if your policy enumerates every external destination.
Also allow outbound HTTPS to your container registry (or mirror), your LLM
upstreams, and your PostgreSQL and Redis.
Inbound
Hybrid is not egress-only. Your published TrustGate LLM and MCP entry points must accept inbound HTTPS from:
Scope the rule to your edge or ingress for those two hosts. Nothing needs to
reach TrustGuard, Firewall, the data-plane API, or DataAgent from outside the
cluster.
Two of these carry long-lived streams. If an egress proxy or middlebox reaps
idle connections, config-sync and DataBridge drop and reconnect on that
interval — check its idle timeout. A TLS-intercepting proxy breaks certificate
verification unless its CA reaches the client: put it in the bundle you point
dataagent.databridge.tlsCa, <product>.configSync.tlsCa and
global.clickstack.egress.tlsCaSecretName at, since each of those replaces the
system roots rather than adding to them.
Install
Step 3 is where most installs fail. Those four Secrets are never generated by
the chart. If your values file does not reference them, the install stops at
render time with a validation error.
1
Create the namespace and image pull secret
global.imageRegistry. Both paths, and the two collector
images that imageRegistry does not rewrite, are in
Container images.2
Create the gateway in the console
Open TrustGate → New Gateway, name it, choose Private, then
Kubernetes. If you are also running TrustGuard, create a private TrustGuard
in TrustGuard → Agent Runtime as well — each product is a separate console
object with its own credentials.
Console setup covers the wizard screen
by screen.The wizard produces a
values.yaml with credentials inline. Treat that file as
a secret, do not commit it, and take exactly two values out of it per product:CONFIG_SYNC_TOKEN— proves to the SaaS control plane that this data plane is yours- the DataAgent enrollment JWT — carries your tenant identity
global.products.agentgateway: true, which the chart rejects. global.products
is keyed by product id, so TrustGate is trustgate there even though its values
block is agentgateway: — see the
naming map.3
Create the four operator-supplied Secrets
Those four values are all you supply:Everything else — JWT signing secrets on both sides of every internal call,
database passwords for in-cluster stores, the config-sync cache key, the MCP
OAuth signing key — is generated on first install and reused on upgrade. Your
own short list is the registry pull secret, these four tokens, and the
credentials for any datastore you provide. See
Secrets.
The enrollment JWT already carries
tenant_id and instance_id. Do not set a
tenant ID in values.4
Write your values file
The chart ships For a subset of products, drop the flags you do not need and use the matching
tracked slice —
values-required.yaml, a full-hybrid preset that already
matches the Secret names above. Copy it and change the two cluster-specific
lines:values-trustgate.yaml.example,
values-trustguard.yaml.example, or values-red-teaming.yaml.example
(data-plane API only, which needs no DataAgent and no config-sync). Every switch
is in
Values cheat sheet.Do not set configSync.enabled: true. Hybrid derives it from the deployment
mode, and restating it is a frequent cause of confusing errors.5
Install
The chart registry is public — no
helm registry login and no credentials
needed to pull it. Only the container images are private, which is what the
gcr-secret pull Secret in step 1 is for. Always pass --version so the
release is reproducible.CreateContainerConfigError
twenty minutes later. The messages name the value to set. To inspect manifests
first:Verify
Running is not the same as working. A Hybrid data plane’s readiness probe includes a snapshot check, so it reports Ready only after it has pulled configuration from SaaS.agentgateway-proxy,
agentgateway-mcp, trustguard-data-plane, data-plane-api, firewall and
its workers, dataagent, and dataagent-trustguard. In-cluster
control-plane-postgresql and redis appear only while deploy: true.
A pod that is Running but never Ready is the signature of config-sync
failing. Three causes, in order of likelihood:
- The token is wrong, or was pasted with trailing whitespace
- Egress to
*.neuraltrust.ai:443is blocked — see Network - A TLS-intercepting proxy is present and its CA is not trusted
configSync.tlsCa. Do not reach for
configSync.tlsInsecure — the runtimes refuse it under a deployed APP_ENV.
Config sync covers the mechanism and its
diagnostics.
Expose both entry points
Hostnames derive fromglobal.domain:
TrustGate listens on two ports for two different protocols, and they need
separate hostnames in production:
What “metadata only” means concretely
Raw prompts and responses are written to your PostgreSQL. What leaves your cluster is telemetry: request metadata, timings, detector verdicts, and counts. The egress path is worth knowing. Aclickstack-egress-collector runs alongside
your primary DataAgent. It exchanges the DataAgent enrollment JWT for a
short-lived OTLP access token through a loopback broker on 127.0.0.1:9465,
then exports over OTLP with that token. There is no long-lived bearer token on
the application pods, and the broker only listens on loopback.
This is also why the chart refuses to install with a product enabled but no
DataAgent enrollment configured: without it the egress collector has nothing to
exchange, so telemetry would silently never leave.
Upgrades
envFrom carry no
checksum of the ConfigMap, so a values change that only touches a ConfigMap
updates the ConfigMap without restarting the pods. The change then takes effect
at the next unrelated restart. If you changed something behavioural, restart the
affected deployment yourself:
High availability
Availability is a ladder, not a single design. Climb it only as far as the failure you actually have to survive — each rung costs more to operate than the one below it.
Tier 1 — one cluster across zones
Spread the node pool over three availability zones, run at least two replicas of TrustGate, TrustGuard and Firewall, and use managed PostgreSQL and Redis with automatic failover inside the region. There is no promotion procedure and no DNS work. This is enough for most deployments — start here and stop here unless a regional requirement says otherwise.Tier 2 — twin clusters in one region
Two clusters side by side, both pointed at one PostgreSQL primary and one Redis primary. Because they are in the same region, sharing both stores adds no network hop and loses nothing on a switch: state is identical from either side. It buys you an escape from a broken cluster or a bad upgrade — roll the second cluster, move traffic, and keep the first as your way back.Tier 3 — two regions, active/passive
The consequence is that rate limits are counted per region. With one active
region at a time that is invisible, except in the switchover window, when a
client could briefly get a fresh allowance. If you need a globally exact counter
across two simultaneously active regions, that is a different design — talk to
us before building it.
Both clusters use the same gateway scope and the same credentials for it. The
passive cluster is warm: its workloads run, pass health checks, and keep
synchronizing, but it receives no production traffic.
What each failure looks like
Running pods survive a control-plane outage from their in-memory configuration,
but a pod that restarts during one needs the last-known-good file on disk.
The chart mounts it on an
emptyDir, which does not survive a restart, so a
restarted pod with no control-plane connectivity will not become Ready. If a
cluster has to tolerate restarts mid-outage, move that mount to persistent
storage. The CONFIG_SYNC_LKG_KEY that encrypts it is generated per cluster and
needs no distribution.
Promote the passive cluster
1
Detect and fence
Use regional health checks backed by Kubernetes readiness on both request paths.
Remove the failed cluster from both global endpoints and stop its DataAgent, so
it cannot come back mid-recovery and register a second stream.
2
Point PostgreSQL at a writable primary
If the primary is in the failed region, promote the read replica and update
global.postgresql.host in the surviving cluster. If the primary is unaffected,
there is nothing to do here — this is the case the single-primary design is
buying you. Redis needs no attention either way, because the surviving cluster
already has its own.3
Verify configuration and start active-only services
Confirm TrustGate and TrustGuard are Ready with a current or last-known-good
configuration, then enable DataAgent. Only one DataAgent may be active.
4
Switch both traffic endpoints
Route the global LLM/proxy and MCP URLs to the promoted cluster. Keep both
protocols on the same cluster, and remember that DNS-based promotion is delayed
by resolver and client caching.
5
Verify enforcement
Run representative allowed and blocked requests, then confirm policy decisions
and metadata export before declaring the failover complete.
Before you rely on it
- Both clusters are deployed, health checked, and pulling the same gateway scope.
- TrustGate and TrustGuard have redundant replicas across zones in each cluster.
- PostgreSQL is managed, with one primary and a cross-region read replica.
- Each region has its own managed Redis, and you accept per-region counters.
- Both cluster edges accept the NeuralTrust inbound source IP.
- Last-known-good configuration is on persistent storage.
- Only the active cluster runs DataAgent.
- The global LLM and MCP URLs resolve to exactly one cluster.
- You have rehearsed: block egress to the control plane and confirm traffic still flows; restart a pod during that outage; promote the passive cluster and switch both URLs.
Next steps
Console setup
Create both products and map their tokens to chart Secrets.
Configuration
Managed stores, ingress, TLS, and every values switch.
Secrets
What the chart generates and what you must supply.
Cloud notes
Ingress, certificates, and managed stores per provider.
Troubleshooting
Install, pod, config-sync, and telemetry failures.