Kafka Exporter
The kafka
telemetry exporter enables TrustGate to emit real-time telemetry events to an Apache Kafka topic. This includes both request-level traces and plugin execution metrics, which are serialized into structured JSON messages and delivered using a Kafka producer.
This exporter allows seamless integration with downstream consumers, including observability platforms, security analytics pipelines, and operational dashboards.
Configuration Example
Configuration Parameters
To activate the Kafka exporter, include it in the gateway’s telemetry section:
Parameter | Type | Required | Description |
---|---|---|---|
host | string | Yes | Kafka broker host (e.g., "localhost" ). |
port | string | Yes | Kafka broker port (e.g., "9092" ). |
topic | string | Yes | Kafka topic where telemetry messages will be published. |
Field Descriptions
enable_request_traces
Enables top-level request tracing. Each incoming request will emit a single summary event containing:
- Latency (
latency_ms
) - Request and response headers
- HTTP method and status code
- Device and client metadata (when available)
This provides end-to-end visibility into the request lifecycle.
enable_plugin_traces
Enables plugin-level tracing. Each plugin executed during the request lifecycle will produce a separate telemetry event containing:
- Input and output values for the plugin
- Error messages or failure states (if any)
- Execution timing and plugin identifier
This allows for detailed diagnostics and plugin performance monitoring.
extra_params
A flexible key-value object that appends static metadata to every telemetry event emitted—both request and plugin traces.
For example:
This configuration results in all Kafka events including the field "app_id": "12345"
in their payloads.
This is useful for:
-
Multi-tenant tracing Distinguish metrics between different applications, clients, or tenants.
-
Environment or region tagging Identify the source environment (e.g., staging, production) or deployment region.
-
Custom labeling for analytics or filtering Enhance downstream processing by attaching meaningful, queryable metadata to every event.
Kafka Message Format
Each Kafka message represents a telemetry event associated with a request processed by TrustGate. The structure differs slightly based on whether the event is a plugin execution trace or a request trace, but both share a common schema foundation.
Plugin Metrics Event
This message is emitted for each plugin executed during a request:
Request-Level Trace Event
Note: Both event types share the same trace_id, allowing downstream systems to correlate plugin metrics with the parent request.
How It Works
Initialization
A Kafka producer is initialized using the specified broker address (host:port
) and target topic. This setup prepares the exporter to begin sending telemetry events as soon as they are generated.
Message Construction
Each telemetry event is serialized into a JSON-formatted message. These messages commonly include fields such as:
trace_id
latency_ms
method
status_code
headers
input
/output
- Device context (e.g.,
device
,os
,browser
)
Message Delivery
Messages are published asynchronously to the configured Kafka topic. A delivery channel is used to confirm whether the message was successfully received or if a transmission error occurred.
Failure Tolerance
The exporter includes error handling for all stages of message transmission:
- If the Kafka producer is uninitialized or has been closed