From RAG

Generate functional tests directly from your RAG knowledge base to validate that your model correctly retrieves and synthesizes information from your documents.

Overview

Testing RAG applications requires validating that:

Retrieval works correctly: Relevant documents are found
Synthesis is accurate: Information is correctly combined
Responses are grounded: Answers are based on the knowledge base
No hallucinations: Model doesn’t make up information

How It Works

TrustTest automatically:

Connects to your knowledge base (vector store, database, etc.)
Retrieves document chunks
Generates question-answer pairs based on the content
Creates test cases with expected responses
Evaluates your model’s actual responses against expectations

Supported Knowledge Bases

Connector	Description
In-Memory	Local vector store for testing
Azure AI Search	Azure’s cognitive search
Neo4j	Graph database
PostgreSQL + pgvector	PostgreSQL with vector extension
Upstash	Serverless Redis vector store

Code Example

Using In-Memory Knowledge Base

from trusttest.knowledge_base import InMemoryKnowledgeBase, Document
from trusttest.probes.rag import RAGProbe
from trusttest.targets.http import HttpTarget, PayloadConfig
from trusttest.evaluators import CorrectnessEvaluator
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluation_scenarios import EvaluationScenario

documents = [
    Document(id="1", content="TrustTest is a framework for testing AI models for safety and reliability.", topic="product"),
    Document(id="2", content="TrustTest supports Azure, Neo4j, PostgreSQL, and Upstash knowledge bases.", topic="connectors"),
    Document(id="3", content="Probes in TrustTest generate test cases to evaluate model behavior.", topic="probes"),
]

kb = InMemoryKnowledgeBase(documents=documents)

# Configure target
target = HttpTarget(
    url="https://your-rag-endpoint.com/chat",
    headers={"Content-Type": "application/json"},
    payload_config=PayloadConfig(
        format={"messages": [{"role": "user", "content": "{{ test }}"}]},
        message_regex="{{ test }}",
    ),
)

# Create RAG probe
probe = RAGProbe(
    target=target,
    knowledge_base=kb,
    num_questions=20,
)

# Evaluate with correctness judge
evaluator = CorrectnessEvaluator()
suite = EvaluatorSuite(evaluators=[evaluator])
scenario = EvaluationScenario(evaluator_suite=suite)

test_set = probe.get_test_set()
results = scenario.evaluate(test_set)
results.display_summary()

Using Azure AI Search

from azure.core.credentials import AzureKeyCredential
from trusttest.knowledge_base.azure_search import AzureKnowledgeBase

kb = AzureKnowledgeBase(
    credentials=AzureKeyCredential("your-api-key"),
    service_endpoint="https://your-search.search.windows.net",
    index_name="your-index",
)

probe = RAGProbe(
    target=target,
    knowledge_base=kb,
    num_questions=50,
)

Using PostgreSQL with pgvector

from trusttest.knowledge_base.pgvector import PgVectorKnowledgeBase

kb = PgVectorKnowledgeBase(
    connection_string="postgresql://user:pass@localhost/db",
    table_name="documents",
    fields_mapping={
        "id": "id",
        "content": "content",
        "embeddings": "embedding",
    },
)

probe = RAGProbe(
    target=target,
    knowledge_base=kb,
    num_questions=50,
)

Configuration Options

Parameter	Type	Default	Description
`target`	`Target`	Required	The RAG model to test
`knowledge_base`	`KnowledgeBase`	Required	Your knowledge base connector
`num_questions`	`int`	`20`	Number of test questions to generate
`question_types`	`List[str]`	`["factual", "inferential"]`	Types of questions to generate
`language`	`LanguageType`	`"English"`	Language for generated questions

Question Types

TrustTest generates different types of questions:

Type	Description	Example
Factual	Direct fact retrieval	”What connectors does TrustTest support?”
Inferential	Requires combining information	”How would you test a RAG app with Azure?”
Comparative	Comparing entities	”What’s the difference between probes and evaluators?”

Evaluating RAG Responses

For RAG applications, use these evaluators:

from trusttest.evaluators import (
    CorrectnessEvaluator,
    CompletenessEvaluator,
    RAGPoisoningEvaluator,
)

evaluators = [
    CorrectnessEvaluator(),      # Is the answer factually correct?
    CompletenessEvaluator(),     # Does it cover all relevant points?
    RAGPoisoningEvaluator(),     # Is the response grounded in context?
]

Gold answers from retrieved docs

Use RagContextBuilder when you want an ExpectedResponseContext dataset grounded in retrieved chunks:

from trusttest.dataset_builder import RagContextBuilder, QuestionTopic

builder = RagContextBuilder(questions=[QuestionTopic(question="...", topic="...")], knowledge_base=kb)
dataset = builder.build()

See Automatic test generation. Configure embeddings and topic_summarizer with set_config before clustering unlabeled documents. Clustering (UMAP / HDBSCAN) is not Azure-only.

Start here

Connect

Concepts

Red teaming

Functional tests

Knowledge bases

Evaluate

Overview

How It Works

Supported Knowledge Bases

Code Example

Using In-Memory Knowledge Base

Using Azure AI Search

Using PostgreSQL with pgvector

Configuration Options

Question Types

Evaluating RAG Responses

Gold answers from retrieved docs

​Overview

​How It Works

​Supported Knowledge Bases

​Code Example

​Using In-Memory Knowledge Base

​Using Azure AI Search

​Using PostgreSQL with pgvector

​Configuration Options

​Question Types

​Evaluating RAG Responses

​Gold answers from retrieved docs

​Related Topics

Overview

How It Works

Supported Knowledge Bases

Code Example

Using In-Memory Knowledge Base

Using Azure AI Search

Using PostgreSQL with pgvector

Configuration Options

Question Types

Evaluating RAG Responses

Gold answers from retrieved docs

Related Topics