Run Responsibility Evaluation

In this guide we will see how to configure and run a Compliance Scenario to evaluate the responsibility and safety of your LLM outputs.

​

Configure the Compliance Scenario

For this example, we’ll create a basic compliance scenario that checks for toxicity in LLM responses.

from dotenv import load_dotenv
from trusttest.catalog import ComplianceScenario
from trusttest.models.testing import DummyEndpoint

load_dotenv()

scenario = ComplianceScenario(
    model=DummyEndpoint(),
    categories={"toxicity"},
    max_objectives_per_category=1,
    use_jailbreaks=False,
)

Let’s break down the configuration parameters:

• model: The LLM model you want to evaluate (in this case, a dummy endpoint for demonstration)
• categories: Set of categories to evaluate.
• max_objectives_per_category: Maximum number of test objectives to generate per category
• use_jailbreaks: Whether to include jailbreak attempts in the evaluation

​

Run the Evaluation

Once you have configured your compliance scenario, you can run the evaluation with these simple steps:

test_set = scenario.probe.get_test_set()
results = scenario.eval.evaluate(test_set)
results.display()
results.display_summary()

This will:

1. Generate a test set based on your configuration
2. Run the evaluation
3. Display detailed results
4. Show a summary of the evaluation

​

Complete Example

Here’s a complete example that puts everything together:

from dotenv import load_dotenv
from trusttest.catalog import ComplianceScenario
from trusttest.models.testing import DummyEndpoint

load_dotenv()

# Configure the compliance scenario
scenario = ComplianceScenario(
    model=DummyEndpoint(),
    categories={"toxicity"},
    max_objectives_per_category=1,
    use_jailbreaks=False,
)

# Run the evaluation
test_set = scenario.probe.get_test_set()
results = scenario.eval.evaluate(test_set)
results.display()
results.display_summary()