Skip to main content

An evaluator returns pass or fail for a test case. Combine them in an evaluator suite, bind the suite to a scenario, and rerun the same plan after every change.

Evaluation strategy

How suites combine evaluators and when a scenario fails.

LLM as judge

Correctness, completeness, tone, relevance, and custom judges.

Heuristics

BLEU, equals, regex, language, signatures, and bias comparison. No extra LLM call.

Concepts

How evaluators sit next to probes, test cases, and scenarios.
Use a heuristic when the check is exact (language, signature, string match). Use an LLM judge when the check is semantic (correctness, tone, relevance). Most catalog threat probes default to TrueFalseEvaluator.