In this guide we will see how to save and load EvaluationScenario, TestSet and EvaluationTestResult.

Client

To track all our evaluation result we use the trusttest.client. Currently, we support two client types:

file-system: Save the results in the local filesystem.
neuraltrust: Save the results in the remote NeuralTrust server.

To define the client:

import os
import trusttest

client = trusttest.client(
    type="neuraltrust",
    token=os.getenv("NEURALTRUST_TOKEN"),
    target_id=os.getenv("NEURALTRUST_TARGET_ID"),
)

import trusttest

client = trusttest.client()
# or
client = trusttest.client(type="file-system")

For the neuraltrust client, target_id is required so saved scenarios, test sets, and runs are scoped to the right NeuralTrust target. You can pass it directly or load it from NEURALTRUST_TARGET_ID.

Save a scenario results

First we need to define our scenario, run the evaluation and get the results.

from trusttest.evaluation_contexts import ExpectedResponseContext
from trusttest.evaluation_scenarios import EvaluationScenario
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluators import BleuEvaluator
from trusttest.targets.testing import DummyTarget
from trusttest.probes.dataset import DatasetProbe
from trusttest.dataset_builder import Dataset, DatasetItem


target = DummyTarget()
probe = DatasetProbe(
    target=target,
    dataset=Dataset([
        [
            DatasetItem(
                question="What is Python?",
                context=ExpectedResponseContext(
                    expected_response="Python is a high-level, interpreted programming language."
                ),
            )
        ]
    ]),
)

test_set = probe.get_test_set()
scenario = EvaluationScenario(
    name="Quickstart Functional Test",
    description="Functional test example.",
    evaluator_suite=EvaluatorSuite(
        evaluators=[BleuEvaluator(threshold=0.3)],
        criteria="any_fail",
    ),
)

results = scenario.evaluate(test_set)

Then we can save the evaluation scenario with just one line:

client.save_evaluation_scenario(scenario)

To save the scenario TestSet:

client.save_evaluation_scenario_test_set(scenario.id, test_set)

Finally, to save the EvaluationTestResult:

client.save_evaluation_scenario_run(results)

Got to your NeuralTrust dashboard to see the results. Or if you are using the file-system client, you can see the results in the trusttest_db folder.

Load a scenario results

We can also load any scenario, test set or evaluation test result from the client. And re-run the evaluation, clone it, etc.

loaded_scenario = client.get_evaluation_scenario(scenario.id)
loaded_scenario_result = client.get_evaluation_scenario_run(scenario.id)
loaded_test_set = client.get_evaluation_scenario_test_set(scenario.id)

result = loaded_scenario.evaluate(loaded_test_set)
result.display()

Complete example

import trusttest
import os
from trusttest.evaluation_contexts import ExpectedResponseContext
from trusttest.evaluation_scenarios import EvaluationScenario
from trusttest.evaluator_suite import EvaluatorSuite
from trusttest.evaluators import BleuEvaluator
from trusttest.targets.testing import DummyTarget
from trusttest.probes.dataset import DatasetProbe
from trusttest.dataset_builder import Dataset, DatasetItem

target = DummyTarget()
probe = DatasetProbe(
    target=target,
    dataset=Dataset([
        [
            DatasetItem(
                question="What is Python?",
                context=ExpectedResponseContext(
                    expected_response="Python is a high-level, interpreted programming language."
                ),
            )
        ]
    ]),
)

test_set = probe.get_test_set()
scenario = EvaluationScenario(
    name="Quickstart Functional Test",
    description="Functional test example.",
    evaluator_suite=EvaluatorSuite(
        evaluators=[BleuEvaluator(threshold=0.3)],
        criteria="any_fail",
    ),
)

results = scenario.evaluate(test_set)

client = trusttest.client(
    type="neuraltrust",
    token=os.getenv("NEURALTRUST_TOKEN"),
    target_id=os.getenv("NEURALTRUST_TARGET_ID"),
)

client.save_evaluation_scenario(scenario)
client.save_evaluation_scenario_test_set(scenario.id, test_set)
client.save_evaluation_scenario_run(results)

loaded_scenario = client.get_evaluation_scenario(scenario.id)
loaded_scenario_result = client.get_evaluation_scenario_run(scenario.id)
loaded_test_set = client.get_evaluation_scenario_test_set(scenario.id)

result = loaded_scenario.evaluate(loaded_test_set)
result.display()

Getting Started

Core Concepts

Connect your app

Create tests

Evaluate results

Save and load Scenarios

Client

Save a scenario results

Load a scenario results

Complete example

​Client

​Save a scenario results

​Load a scenario results

​Complete example

Client

Save a scenario results

Load a scenario results

Complete example