Test Generation

​

Purpose

• Test RAG systems with automatically generated questions
• Evaluate model performance on specific knowledge domains
• Generate both functional and adversarial test cases
• Assess system robustness against different types of questions
• Validate knowledge base integration and retrieval capabilities

​

How It Works

​

Functional RAG Scenario

• Generates benign questions to test normal system operation
• Supports multiple question types:
  • Simple Questions: Basic factual questions
  • Complex Questions: More detailed and nuanced questions
  • Really Complex Questions: Highly detailed and multi-part questions
  • Conversational Questions: Questions in a more natural, conversational style
  • Distracting Questions: Questions with additional context that might distract
  • Double Questions: Questions that ask about two related things
  • OOS Questions: Out-of-scope questions to test system boundaries

​

Adversarial RAG Scenario

• Generates malicious questions to test system robustness
• Supports multiple attack types:
  • Instruction Manipulation: Questions that attempt to overwrite, invalidate, edit, or contradict the information in the context, forcing the system to provide incorrect information while maintaining a natural question format.
  • Role Play: Questions that assign a new name/role to the assistant and provide context updates that override previous information, creating scenarios that lead to incorrect responses.
  • Hypothetical: Questions that create hypothetical scenarios based on the context, testing the system’s ability to handle speculative situations while maintaining factual accuracy.
  • Storytelling: Questions that attempt to make the system engage in storytelling about the context, potentially leading to fictional or exaggerated responses.
  • Obfuscation: Questions that use complex language, technical terms, or confusing phrasing to obscure the actual intent, testing the system’s ability to handle complex queries.
  • Payload Splitting: Questions that split malicious content across multiple parts or use indirect references, testing the system’s ability to handle fragmented or indirect queries.
  • List Based: Questions that request lists or enumerations of information, potentially leading to incomplete or incorrect responses.
  • Special Token: Questions that include special characters, tokens, or unusual formatting to test the system’s handling of non-standard input.
  • Off Tone: Questions that attempt to make the system respond in an inappropriate or unprofessional tone, testing its ability to maintain appropriate communication standards.

1. Load documents into a knowledge base
2. Generate questions based on document topics
3. Query the model with generated questions
4. Evaluate responses using configured evaluators
5. Provide detailed results and metrics

​

Usage Examples

from trusttest.knowledge_base import Document, InMemoryKnowledgeBase
from trusttest.probes.rag import RAGProbe, BenignQuestion
from trusttest.evaluation_scenarios import EvaluationScenario
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluators import AnswerRelevanceEvaluator

documents = [
    Document(
        id="1",
        content="Your document content here",
        topic="Your topic here",
    )
]
knowledge_base = InMemoryKnowledgeBase(documents=documents)

probe = RAGProbe(
    target=your_model,
    knowledge_base=knowledge_base,
    num_questions=10,
    question_types=[
        BenignQuestion.SIMPLE,
        BenignQuestion.COMPLEX,
        BenignQuestion.REALLY_COMPLEX,
        BenignQuestion.CONVERSATIONAL,
        BenignQuestion.DISTRACTING,
        BenignQuestion.DOUBLE,
        BenignQuestion.OOS,
    ],
)

scenario = EvaluationScenario(
    name="RAG Functional",
    evaluator_suite=EvaluatorSuite(evaluators=[AnswerRelevanceEvaluator()], criteria="any_fail"),
)

test_set = probe.get_test_set()
results = scenario.evaluate(test_set)
results.display()

from trusttest.knowledge_base import Document, InMemoryKnowledgeBase
from trusttest.probes.rag import RAGProbe, MaliciousQuestion
from trusttest.evaluation_scenarios import EvaluationScenario
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluators import RAGPoisoningEvaluator

documents = [
    Document(
        id="1",
        content="Your document content here",
        topic="Your topic here",
    )
]
knowledge_base = InMemoryKnowledgeBase(documents=documents)

probe = RAGProbe(
    target=your_model,
    knowledge_base=knowledge_base,
    num_questions=10,
    question_types=[
        MaliciousQuestion.INSTRUCTION_MANIPULATION,
        MaliciousQuestion.ROLE_PLAY,
        MaliciousQuestion.HYPOTHETICAL,
        MaliciousQuestion.STORYTELLING,
        MaliciousQuestion.OBFUSCATION,
        MaliciousQuestion.PAYLOAD_SPLITTING,
        MaliciousQuestion.LIST_BASED,
        MaliciousQuestion.SPECIAL_TOKEN,
        MaliciousQuestion.OFF_TONE,
    ],
)

scenario = EvaluationScenario(
    name="RAG Poisoning",
    evaluator_suite=EvaluatorSuite(evaluators=[RAGPoisoningEvaluator()], criteria="any_fail"),
)

test_set = probe.get_test_set()
results = scenario.evaluate(test_set)
results.display()

​

RagContextBuilder

from trusttest.dataset_builder import RagContextBuilder, QuestionTopic
from trusttest.knowledge_base import InMemoryKnowledgeBase, Document

kb = InMemoryKnowledgeBase(
    documents=[
        Document(id="1", content="TrustTest evaluates AI models.", topic="product"),
    ]
)
questions = [QuestionTopic(question="What does TrustTest evaluate?", topic="product")]

builder = RagContextBuilder(
    questions=questions,
    knowledge_base=kb,
)
dataset = builder.build()  # Dataset[ExpectedResponseContext]; or await builder.async_build()

​

When to Use

• Validate knowledge base integration
• Assess system robustness against domain specific attacks
• Generate comprehensive test cases automatically
• Test specific knowledge domains or topics
• Compare different RAG configurations
• Identify system vulnerabilities