Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.neuraltrust.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
What this covers

Any application or service that calls an LLM provider directly through an official SDK. Because you control the call, the Gateway sits in between as an HTTPS proxy and applies policies on both the request and the response.
  • Surface: Gateway
  • Who is this for: backends and scripts using openai, anthropic, @google/generative-ai, Azure OpenAI, the Bedrock Runtime SDK, or the Vercel AI SDK.

​
Architecture

your app ──► TrustGate Gateway ──► LLM provider (OpenAI, Anthropic, Google, Azure, Bedrock)
                │
                └── applies route matching, policies, detectors, observability
The Gateway speaks the provider’s native wire format, so your SDK doesn’t need any provider-specific changes beyond the base URL and API key.

​
Step-by-step setup

Setup happens entirely in the Integrations area of the console. The provider’s native credentials (OpenAI API key, Azure OpenAI endpoint + key, AWS credentials, etc.) live on a provider Integration — never on your client and never on a Route. The Route just picks that Integration from a dropdown.
1

Register the LLM provider as an Integration

Open Integrations → Add Integration, pick your provider from the catalog (OpenAI, Anthropic, Azure OpenAI, Google, Bedrock, Mistral, Cohere, …), and enter the provider’s credentials:
  • OpenAI / Anthropic / Mistral / Cohere / Google / Groq / Together — provider API key.
  • Azure OpenAI — resource endpoint, API key, and default API version.
  • AWS Bedrock — IAM role ARN, access key pair, or workload identity. The region and the models you want to expose.
  • Self-hosted model — the internal base URL and any auth header.
Give the Integration a short name (for example openai-prod, bedrock-eu-west-1). Repeat for every provider you want to front.
2

Create a Gateway Integration

Still in Integrations → Add Integration, pick Gateway. Choose Serverless (instant, good for dev) or Dedicated (production), give it a name (prod-eu, staging), and optionally add Tags.After saving, open Gateway → Overview and copy the Endpoint — it looks like https://<gateway>.neuraltrust.ai/v1. This is the only URL your client needs.Default Routes for every provider Integration you registered in Step 1 are created automatically on this Gateway. You can inspect them under Gateway → Routes, add Use Cases or Tags for policy scoping, or set per-route rate limits — but no manual Route creation is needed to start serving traffic.
3

Issue a Gateway API key

Open the Gateway Integration’s API Keys tab and create a key. This is the credential your client will use — the provider’s native key stays on the Integration and is never seen by the client.
4

Swap base URL and API key in your client

Point the SDK at https://<gateway>.neuraltrust.ai/v1 and use the TrustGate key. No other changes — paths, method, headers, body, and streaming are unchanged. See the language-specific snippets below.
5

Verify in Runtime → Explorer

Send one request from your app and confirm it appears in Runtime → Explorer with the matching Route, any detector decisions, and token usage.

​
Client code

The client never sees the provider’s API key. The only change is the base URL and the API key you pass to the SDK.

​
OpenAI (Python)

from openai import OpenAI

client = OpenAI(
    base_url="https://<gateway>.neuraltrust.ai/v1",
    api_key="<trustgate-api-key>",
)

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

​
OpenAI (TypeScript)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://<gateway>.neuraltrust.ai/v1",
  apiKey: "<trustgate-api-key>",
});

​
Anthropic (Python)

from anthropic import Anthropic

client = Anthropic(
    base_url="https://<gateway>.neuraltrust.ai",
    api_key="<trustgate-api-key>",
)

​
Azure OpenAI

Use an Azure OpenAI route on the Gateway and point the Azure SDK at it: 
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://<gateway>.neuraltrust.ai",
    api_key="<trustgate-api-key>",
    api_version="2024-10-21",
)

​
AWS Bedrock (via OpenAI-compatible route)

For direct model invocation on Bedrock (not the managed Bedrock Agents service), the Gateway speaks OpenAI to the client and translates to Bedrock upstream — so you can use the standard OpenAI SDK with a Bedrock model ID. No boto3, no SigV4 on the client. 
from openai import OpenAI

client = OpenAI(
    base_url="https://<gateway>.neuraltrust.ai/v1",
    api_key="<trustgate-api-key>",
)

resp = client.chat.completions.create(
    model="anthropic.claude-sonnet-4-20250514-v1:0",
    messages=[{"role": "user", "content": "Hello"}],
)
AWS credentials live on the Bedrock Integration you registered in Step 1. For agents using the Strands SDK on top of this pattern, see AWS Bedrock Agents.

​
Vercel AI SDK

import { createOpenAI } from "@ai-sdk/openai";

const openai = createOpenAI({
  baseURL: "https://<gateway>.neuraltrust.ai/v1",
  apiKey: "<trustgate-api-key>",
});

​
Policies to apply

Policies on this integration use the Gateway surface and follow the standard Where / When / Then model — see Policies & Enforcement for the authoring flow and precedence rules. For a direct-SDK integration, scope policies with the Gateways or Routes filters so rules apply to the specific provider route (OpenAI, Anthropic, Bedrock, Azure OpenAI, etc.). Start with these five patterns, each authored in Log mode first and promoted to Mask / Block once you’ve reviewed hits in Runtime → Logs.

​
Block prompt injection and jailbreaks on user input

  • Where — Gateway + filter Gateways = <your-gateway>
  • When — Input · Triggers · Prompt Injection, Jailbreak
  • Then — Block

​
Mask PII on both sides of the hop

  • Where — Gateway + filter Gateways = <your-gateway>
  • When — Input or Output · Triggers · Email Address, Phone Number, Credit Card, Social Security Number
  • Then — Mask

​
Block secret / credential leakage

  • Where — Gateway + filter Gateways = <your-gateway>
  • When — Input or Output · Triggers · API Key / Secret
  • Then — Block

​
Moderate toxic or harmful outputs

  • Where — Gateway + filter Routes = <customer-facing-routes>
  • When — Output · Triggers · Toxicity, Harmful Content
  • Then — Block

​
Enforce per-consumer token and request limits

  • Where — Gateway + filter Gateways = <your-gateway>
  • When — API Key · Exceeds · <rate-limit-threshold>
  • Then — Block
For tool-calling flows (tools=[...] on the SDK), layer on a tool-call guard policy that matches on the Tool Call field to validate arguments the model emits before downstream systems act on them.

​
Limitations

  • Streaming: requests using stream=true are inspected as a full response once the stream completes. Mask and Block actions take effect on the final consolidated payload, not per chunk.
  • Provider-specific payloads: each SDK targets its provider’s native shape. Make sure the Route type matches the SDK (OpenAI-compatible, Anthropic, Bedrock, Azure OpenAI, etc.).
  • Conversation context: each call is inspected independently. To link turns in multi-turn conversations, send a stable conversation_id header so Explorer can thread them.
  • Native tool calling: tool-call arguments and outputs are inspected at the LLM boundary. Deeper inspection of downstream tool execution requires separately wrapping those tools with a Gateway or API integration.