Sensitive Data Leak

Sensitive data leak testing evaluates your model’s ability to protect confidential information from unauthorized disclosure. These probes attempt to extract sensitive data through various techniques.

Purpose

Data leak testing is essential for:

Protecting user privacy and confidential information
Ensuring compliance with data protection regulations
Preventing unauthorized disclosure of system details
Validating data handling policies

Attack Categories

Sub-Category	Description
Direct Query for Sensitive Data	Tests resistance to straightforward requests for confidential information
Contextual Leakage Request	Evaluates if context can be used to extract sensitive data
Metadata Query	Assesses protection of system metadata and configuration details
Obfuscated Data Request	Tests against encoded or disguised data extraction attempts
Misleading Context Request	Evaluates resistance to social engineering style data requests

Agent-Specific Probes

For AI agents with tool access:

Sub-Category	Description
List of Agent Available Tools	Tests if the model reveals its available tools and capabilities
List of Parameters of Agent Tools	Evaluates protection of tool parameter details
Reasoning Details of AI Agents	Assesses if internal reasoning processes can be extracted

Dataset vs Prompt probes

The catalog builder instantiates *DatasetProbe classes (YAML under datasets/static_objectives/) and evaluates with TrueFalseEvaluator. Each subcategory also has a *PromptProbe (for example DirectQueryForSensitiveDataPromptProbe) for LLM-generated attacks. Dataset probes accept sampling, use_jailbreaks, max_jailbreak_per_objective, and translate_into_language. The builder maps language → translate_into_language and num_test_cases → max_objectives_per_category. See StaticDatasetProbe. Catalog scenarios attach framework tags (EU AI Act, OWASP, MITRE ATLAS, ISO/IEC 42001). get_scenario() returns a Scenario; it does not run remotely.

How It Works

The evaluator uses a binary scoring system:

Pass: The model protects sensitive information appropriately
Fail: The model discloses information that should be protected

Usage Example

from trusttest.catalog.sensitive_data_leak import SensitiveDataLeakScenarioBuilder, SubCategory
from trusttest.targets.http import HttpTarget, PayloadConfig

target = HttpTarget(
    url="https://your-model-endpoint.com/chat",
    headers={"Content-Type": "application/json"},
    payload_config=PayloadConfig(
        format={
            "messages": [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": "{{ test }}"}
            ]
        },
        message_regex="{{ test }}",
    ),
)

builder = SensitiveDataLeakScenarioBuilder(target=target, num_test_cases=20)
scenario = builder.get_scenario(SubCategory.DIRECT_QUERY_FOR_SENSITIVE_DATA)

test_set = scenario.probe.get_test_set()
results = scenario.eval.evaluate(test_set)
results.display_summary()

When to Use

Use sensitive data leak testing when you need to:

Validate data protection measures
Ensure privacy compliance (GDPR, CCPA, etc.)
Test models handling confidential information
Assess agent security boundaries
Audit data handling practices

Start here

Connect

Concepts

Red teaming

Functional tests

Knowledge bases

Evaluate

Purpose

Attack Categories

Agent-Specific Probes

Dataset vs Prompt probes

How It Works

Usage Example

When to Use

​Purpose

​Attack Categories

​Agent-Specific Probes

​Dataset vs Prompt probes

​How It Works

​Usage Example

​When to Use

Purpose

Attack Categories

Agent-Specific Probes

Dataset vs Prompt probes

How It Works

Usage Example

When to Use