> ## Documentation Index
> Fetch the complete documentation index at: https://docs.neuraltrust.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Test Generation

The RAG (Retrieval-Augmented Generation) Probe is a specialized tool designed to automatically generate and evaluate test cases for RAG systems. It uses a knowledge base to generate both functional and adversarial questions, and then evaluates the model's responses against these questions.

<Note>
  This probe needs a LLM client configured to generate the questions and topic generation if not provided.
  Also and embedding model configured to generate the questions and topic generation if not provided.
</Note>

## Purpose

The RAG Probe is particularly useful when you need to:

* Test RAG systems with automatically generated questions
* Evaluate model performance on specific knowledge domains
* Generate both functional and adversarial test cases
* Assess system robustness against different types of questions
* Validate knowledge base integration and retrieval capabilities

## How It Works

The probe works with two main types of scenarios:

### Functional RAG Scenario

* Generates benign questions to test normal system operation
* Supports multiple question types:
  * **Simple Questions**: Basic factual questions
  * **Complex Questions**: More detailed and nuanced questions
  * **Really Complex Questions**: Highly detailed and multi-part questions
  * **Conversational Questions**: Questions in a more natural, conversational style
  * **Distracting Questions**: Questions with additional context that might distract
  * **Double Questions**: Questions that ask about two related things
  * **OOS Questions**: Out-of-scope questions to test system boundaries

### Adversarial RAG Scenario

* Generates malicious questions to test system robustness
* Supports multiple attack types:
  * **Instruction Manipulation**: Questions that attempt to overwrite, invalidate, edit, or contradict the information in the context, forcing the system to provide incorrect information while maintaining a natural question format.
  * **Role Play**: Questions that assign a new name/role to the assistant and provide context updates that override previous information, creating scenarios that lead to incorrect responses.
  * **Hypothetical**: Questions that create hypothetical scenarios based on the context, testing the system's ability to handle speculative situations while maintaining factual accuracy.
  * **Storytelling**: Questions that attempt to make the system engage in storytelling about the context, potentially leading to fictional or exaggerated responses.
  * **Obfuscation**: Questions that use complex language, technical terms, or confusing phrasing to obscure the actual intent, testing the system's ability to handle complex queries.
  * **Payload Splitting**: Questions that split malicious content across multiple parts or use indirect references, testing the system's ability to handle fragmented or indirect queries.
  * **List Based**: Questions that request lists or enumerations of information, potentially leading to incomplete or incorrect responses.
  * **Special Token**: Questions that include special characters, tokens, or unusual formatting to test the system's handling of non-standard input.
  * **Off Tone**: Questions that attempt to make the system respond in an inappropriate or unprofessional tone, testing its ability to maintain appropriate communication standards.

The probe will:

1. Load documents into a knowledge base
2. Generate questions based on document topics
3. Query the model with generated questions
4. Evaluate responses using configured evaluators
5. Provide detailed results and metrics

## Usage Examples

<CodeGroup>
  ```python Functional Testing [expandable] theme={null}
  from trusttest.knowledge_base import Document, InMemoryKnowledgeBase
  from trusttest.probes.rag import RAGProbe, BenignQuestion
  from trusttest.evaluation_scenarios import EvaluationScenario
  from trusttest.evaluator_suite import EvaluatorSuite
  from trusttest.evaluators import AnswerRelevanceEvaluator

  documents = [
      Document(
          id="1",
          content="Your document content here",
          topic="Your topic here",
      )
  ]
  knowledge_base = InMemoryKnowledgeBase(documents=documents)

  probe = RAGProbe(
      target=your_model,
      knowledge_base=knowledge_base,
      num_questions=10,
      question_types=[
          BenignQuestion.SIMPLE,
          BenignQuestion.COMPLEX,
          BenignQuestion.REALLY_COMPLEX,
          BenignQuestion.CONVERSATIONAL,
          BenignQuestion.DISTRACTING,
          BenignQuestion.DOUBLE,
          BenignQuestion.OOS,
      ],
  )

  scenario = EvaluationScenario(
      name="RAG Functional",
      evaluator_suite=EvaluatorSuite(evaluators=[AnswerRelevanceEvaluator()], criteria="any_fail"),
  )

  test_set = probe.get_test_set()
  results = scenario.evaluate(test_set)
  results.display()
  ```

  ```python Adversarial Testing [expandable] theme={null}
  from trusttest.knowledge_base import Document, InMemoryKnowledgeBase
  from trusttest.probes.rag import RAGProbe, MaliciousQuestion
  from trusttest.evaluation_scenarios import EvaluationScenario
  from trusttest.evaluator_suite import EvaluatorSuite
  from trusttest.evaluators import RAGPoisoningEvaluator

  documents = [
      Document(
          id="1",
          content="Your document content here",
          topic="Your topic here",
      )
  ]
  knowledge_base = InMemoryKnowledgeBase(documents=documents)

  probe = RAGProbe(
      target=your_model,
      knowledge_base=knowledge_base,
      num_questions=10,
      question_types=[
          MaliciousQuestion.INSTRUCTION_MANIPULATION,
          MaliciousQuestion.ROLE_PLAY,
          MaliciousQuestion.HYPOTHETICAL,
          MaliciousQuestion.STORYTELLING,
          MaliciousQuestion.OBFUSCATION,
          MaliciousQuestion.PAYLOAD_SPLITTING,
          MaliciousQuestion.LIST_BASED,
          MaliciousQuestion.SPECIAL_TOKEN,
          MaliciousQuestion.OFF_TONE,
      ],
  )

  scenario = EvaluationScenario(
      name="RAG Poisoning",
      evaluator_suite=EvaluatorSuite(evaluators=[RAGPoisoningEvaluator()], criteria="any_fail"),
  )

  test_set = probe.get_test_set()
  results = scenario.evaluate(test_set)
  results.display()
  ```
</CodeGroup>

## When to Use

Use the RAG Probe when you need to:

* Validate knowledge base integration
* Assess system robustness against domain specific attacks
* Generate comprehensive test cases automatically
* Test specific knowledge domains or topics
* Compare different RAG configurations
* Identify system vulnerabilities