Quickstart

Install the package, point a probe at a target, bind evaluators to a scenario, and print the results. This run uses a dummy target so you can confirm the loop before connecting a real model.

uv add trusttest

Jump to the complete example if you want the full script in one block.

Step 1: Target

Define the system under test.

DummyTarget ships with TrustTest so you can exercise the loop without a live model.

from trusttest.targets.testing import DummyTarget

target = DummyTarget()
response = target.respond("Hello, how are you?")
print(response)

The dummy target returns a fixed reply for a fixed set of inputs. Anything else comes back as “I don’t know the answer to that question.”

Step 2: Probe

Generate the test cases.

DatasetProbe turns a small Q&A dataset into test cases against the target.

from trusttest.dataset_builder import Dataset, DatasetItem
from trusttest.evaluation_contexts import ExpectedResponseContext
from trusttest.targets.testing import DummyTarget
from trusttest.probes.dataset import DatasetProbe

target = DummyTarget()
probe = DatasetProbe(
    target=target,
    dataset=Dataset(
        [
            [
                DatasetItem(
                    question="What is Python?",
                    context=ExpectedResponseContext(
                        expected_response="Python is a high-level, interpreted programming language."
                    ),
                )
            ],
            [
                DatasetItem(
                    question="What is the capital of France?",
                    context=ExpectedResponseContext(
                        expected_response="The capital of France is Paris."
                    ),
                )
            ],
        ]
    ),
)

test_set = probe.get_test_set()

The test_set has two test cases: the question, the model response, and the context used to score it.

Step 3: Scenario

Choose evaluators and failure criteria.

Bind the test set to the metrics that decide pass or fail. any_fail fails the scenario if either evaluator fails.

from trusttest.evaluation_scenarios import EvaluationScenario
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluators import (
    BleuEvaluator,
    ExpectedLanguageEvaluator,
)

scenario = EvaluationScenario(
    name="Quickstart Functional Test",
    description="Functional test example.",
    evaluator_suite=EvaluatorSuite(
        evaluators=[
            BleuEvaluator(threshold=0.3),
            ExpectedLanguageEvaluator(expected_languages="English"),
        ],
        criteria="any_fail",
    ),
)

Step 4: Run

Evaluate the test set and print results.

results = scenario.evaluate(test_set)
results.display()
results.display_summary()

Continue with the local LLM tutorial or connect a real HTTP target.

Complete example

Full Python script for the quickstart.

from trusttest.dataset_builder import Dataset, DatasetItem
from trusttest.evaluation_contexts import ExpectedResponseContext
from trusttest.evaluation_scenarios import EvaluationScenario
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluators import (
    BleuEvaluator,
    ExpectedLanguageEvaluator,
)
from trusttest.targets.testing import DummyTarget
from trusttest.probes.dataset import DatasetProbe

target = DummyTarget()
probe = DatasetProbe(
    target=target,
    dataset=Dataset(
        [
            [
                DatasetItem(
                    question="What is Python?",
                    context=ExpectedResponseContext(
                        expected_response="Python is a high-level, interpreted programming language."
                    ),
                )
            ],
            [
                DatasetItem(
                    question="What is the capital of France?",
                    context=ExpectedResponseContext(
                        expected_response="The capital of France is Paris."
                    ),
                )
            ],
        ]
    ),
)

test_set = probe.get_test_set()

scenario = EvaluationScenario(
    name="Quickstart Functional Test",
    description="Functional test example.",
    evaluator_suite=EvaluatorSuite(
        evaluators=[
            BleuEvaluator(threshold=0.3),
            ExpectedLanguageEvaluator(expected_languages="English"),
        ],
        criteria="any_fail",
    ),
)

results = scenario.evaluate(test_set)
results.display()
results.display_summary()

Start here

Connect

Concepts

Red teaming

Functional tests

Knowledge bases

Evaluate