Functional tests

Functional tests check product quality, not security. Use them next to red teaming when you need both evidence in one plan.

Test generation methods

From RAG

Generate tests from your knowledge base

From Dataset

Use existing Q&A datasets

From Prompt

Generate tests dynamically with LLMs

When to Use Functional Testing

Use Case	Recommended Approach
RAG applications	From RAG - tests against your actual knowledge base
Customer support bots	From Dataset - curated Q&A pairs
General assistants	From Prompt - dynamic test generation
Domain-specific models	Combination of all approaches

Evaluation Methods

Functional tests can be evaluated using:

LLM-as-Judge: Use an LLM to assess response quality
Heuristics: Use BLEU, exact match, regex patterns
Custom evaluators: Define your own evaluation logic

Learn more about evaluation →

Quick Example

from trusttest.probes.rag import RAGProbe
from trusttest.evaluation_scenarios import EvaluationScenario
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluators import AnswerRelevanceEvaluator
from trusttest.targets.http import HttpTarget, PayloadConfig

target = HttpTarget(
    url="https://your-model-endpoint.com/chat",
    headers={"Content-Type": "application/json"},
    payload_config=PayloadConfig(
        format={"messages": [{"role": "user", "content": "{{ test }}"}]},
        message_regex="{{ test }}",
    ),
)

probe = RAGProbe(
    target=target,
    knowledge_base=your_knowledge_base,
    num_questions=50,
)

scenario = EvaluationScenario(
    name="RAG Functional",
    evaluator_suite=EvaluatorSuite(evaluators=[AnswerRelevanceEvaluator()], criteria="any_fail"),
)

test_set = probe.get_test_set()
results = scenario.evaluate(test_set)
results.display_summary()

Start here

Connect

Concepts

Red teaming

Knowledge bases

Evaluate

Functional tests

Test generation methods

From RAG

From Dataset

From Prompt

When to Use Functional Testing

Evaluation Methods

Quick Example

​Test generation methods

From RAG

From Dataset

From Prompt

​When to Use Functional Testing

​Evaluation Methods

​Quick Example

Test generation methods

When to Use Functional Testing

Evaluation Methods

Quick Example