Run Evaluations

The 10xgraph eval command discovers evaluation files in your project, runs all cases under a single async event loop, and always generates an HTML and JSON report. No flags required, reports are on by default.

Prerequisites

Your project must have been initialised with 10xgraph init. Eval files live in the evals/ directory, which is generated when you choose the Production setup during 10xgraph init.

Quick start

From the folder that contains 10xgraph.json:

10xgraph eval

This scans evals/ for files matching *_eval.py or eval_*.py, collects every case from every file into a flat pool, runs them, and writes reports to eval_reports/:

eval_reports/
  weather-agent-regression_20260513_142301.html
  weather-agent-regression_20260513_142301.json

Run a specific file or directory

# One file
10xgraph eval evals/weather_agents_eval.py

# A subdirectory
10xgraph eval evals/regression/

When a file is given, only that file runs. When a directory is given, all matching files are discovered. Results from all files are merged into a single combined report.

Run in parallel

By default all cases run sequentially. Pass --parallel to run them concurrently:

10xgraph eval --parallel
10xgraph eval --parallel --max-concurrency 8

How it works: all cases from all files are collected first into a single flat pool. One asyncio event loop runs the entire pool under a single semaphore capped at --max-concurrency. Cases complete out of order, that is expected.

[  1/50] weather_agents_eval.py::weather_london      PASSED   1.23s
[  3/50] booking_eval.py::book_flight_london         PASSED   2.10s
[  2/50] weather_agents_eval.py::weather_new_york    PASSED   0.98s
...
[ 50/50] ...

Results: 47/50 passed (94.0%)

You can also enable parallel by default in 10xgraph.json (see Configure defaults).

Reports

Every run produces two files in eval_reports/ (or the directory set by --output):

Console output is always printed as cases complete. The HTML/JSON files are written after all cases finish.

Open the report automatically

10xgraph eval --open

Skip file output

10xgraph eval --no-report

Only console output is produced. Useful for fast local feedback.

Set a pass-rate threshold

10xgraph eval --threshold 0.8

The command exits with a non-zero code if the overall pass rate is below the threshold. Useful in CI to gate merges on eval quality.

Write reports to a custom directory

10xgraph eval --output ci/reports

Configure defaults in 10xgraph.json

Add an evaluation section to 10xgraph.json to set project-level defaults. CLI flags always take precedence.

{
  "agent": "graph.agent:app",
  "evaluation": {
    "directory": "evals",
    "output_dir": "eval_reports",
    "threshold": 0.75,
    "parallel": false,
    "max_concurrency": 4
  }
}

Report filenames from 10xgraph eval always carry a timestamp; 10xgraph.json has no setting for it.

Enforce threshold in CI

# .github/workflows/ci.yml
- name: Run evaluations
  run: 10xgraph eval --parallel

Set threshold in 10xgraph.json. If the pass rate drops below it, the step fails without extra flags.

Eval file protocols

An eval file is any *_eval.py or eval_*.py file. The CLI auto-detects which protocol you are using. Pick the one that fits your use case.

Summary

get_eval_set(), minimum required

The CLI loads the agent from 10xgraph.json, applies default criteria (60% threshold on all), runs the evaluation, and writes reports. You only define the cases.

# evals/weather_agents_eval.py
from tenxgraph.qa.evaluation import EvalSet, EvalSetBuilder

def get_eval_set() -> EvalSet:
    return (
        EvalSetBuilder(name="weather-agent-regression")
        .add_tool_test(
            query="What is the weather in London?",
            tool_name="get_weather",
            tool_args={"location": "London"},
            expected_response="London",
            case_id="weather_london",
        )
        .add_tool_test(
            query="What is the weather in Tokyo?",
            tool_name="get_weather",
            tool_args={"location": "Tokyo"},
            expected_response="Tokyo",
            case_id="weather_tokyo",
        )
        .build()
    )

Default criteria (applied automatically when no criteria are configured anywhere):

get_eval_config(), per-file criteria with EvalPresets

Add this function when you want to specify which criteria to run and what thresholds to use. The recommended approach is EvalPresets, one-line preset configs covering the most common patterns.

from tenxgraph.qa.evaluation import EvalConfig, EvalSet, EvalSetBuilder
from tenxgraph.qa.evaluation.config.presets import EvalPresets

def get_eval_config() -> EvalConfig:
    return EvalPresets.tool_usage(threshold=0.6)

def get_eval_set() -> EvalSet:
    return (
        EvalSetBuilder(name="weather-agent-regression")
        .add_tool_test(
            query="What is the weather in London?",
            tool_name="get_weather",
            tool_args={"location": "London"},
            expected_response="London",
            case_id="weather_london",
        )
        .build()
    )

Available presets:

You can also combine presets:

from tenxgraph.qa.evaluation.config.presets import EvalPresets

def get_eval_config():
    return EvalPresets.combine(
        EvalPresets.tool_usage(threshold=0.7),
        EvalPresets.response_quality(threshold=0.6),
    )

EVAL_CONFIG, constant instead of function

Same effect as get_eval_config() but as a module-level constant. Useful when the config is static.

from tenxgraph.qa.evaluation.config.presets import EvalPresets

EVAL_CONFIG = EvalPresets.tool_usage(threshold=0.6)

confeval.py, global eval config

Place a file named exactly confeval.py in your project root (next to 10xgraph.json) to set a global default EvalConfig that applies to every eval file that does not define its own get_eval_config() or EVAL_CONFIG. If a file does provide its own config, that takes precedence and confeval.py is ignored for that file.

The file must expose either a module-level EVAL_CONFIG variable or a callable get_eval_config() that returns an EvalConfig.

# confeval.py  (project root, next to 10xgraph.json)
from tenxgraph.qa.evaluation import CriteriaConfig, CriterionConfig, EvalConfig

EVAL_CONFIG = EvalConfig(
    criteria=CriteriaConfig(
        tool_name_match=CriterionConfig.tool_name_match(threshold=1.0),
        # response_match=CriterionConfig.response_match(threshold=0.8),
        # hallucinations=CriterionConfig.hallucination(threshold=0.8),
        rouge_match=CriterionConfig.rouge_match(threshold=0.5),
    )
)

Or as a function:

# confeval.py
from tenxgraph.qa.evaluation import CriteriaConfig, CriterionConfig, EvalConfig

def get_eval_config() -> EvalConfig:
    return EvalConfig(
        criteria=CriteriaConfig(
            tool_name_match=CriterionConfig.tool_name_match(threshold=1.0),
            rouge_match=CriterionConfig.rouge_match(threshold=0.5),
        )
    )

If confeval.py is absent and a file has no per-file config, the built-in defaults apply (all criteria at 0.6 threshold).

Annotated functions -> EvalSet, pytest-style discovery

Any module-level function with return type -> EvalSet is auto-discovered as an eval set. Useful when you want multiple named eval sets in one file.

from tenxgraph.qa.evaluation import EvalSet, EvalSetBuilder
from tenxgraph.qa.evaluation.config.presets import EvalPresets

def get_eval_config():
    return EvalPresets.tool_usage(threshold=0.6)

def weather_cases() -> EvalSet:
    return EvalSetBuilder(name="weather").add_tool_test(...).build()

def booking_cases() -> EvalSet:
    return EvalSetBuilder(name="booking").add_tool_test(...).build()

Both weather_cases and booking_cases are discovered and run. Their results appear as separate eval sets in the report.

get_scenarios(), user simulator

Use this protocol when you want the LLM to drive a dynamic multi-turn conversation against your agent rather than using fixed prompt/response pairs.

You only define the scenarios. The CLI handles running the simulator, scoring goal achievement, and writing the report, identical to regular eval cases.

# evals/user_simulator_eval.py
from tenxgraph.qa.evaluation import ConversationScenario, UserSimulatorConfig

# Optional: override simulator model and settings for this file.
# If omitted, the CLI uses UserSimulatorConfig defaults (gemini-2.5-flash).
SIMULATOR_CONFIG = UserSimulatorConfig(
    model="gemini/gemini-2.5-flash",
    max_invocations=8,
    temperature=0.7,
)

def get_scenarios() -> list[ConversationScenario]:
    return [
        ConversationScenario(
            scenario_id="weather_travel_planning",
            description="User planning a trip wants weather info and packing advice",
            starting_prompt="Hi! I'm planning a trip to Paris this weekend.",
            conversation_plan=(
                "1. Ask about current weather in Paris\n"
                "2. Ask whether to bring a jacket\n"
                "3. Ask about outdoor sightseeing timing"
            ),
            goals=[
                "User receives weather information for Paris",
                "User gets clothing or packing advice",
                "User learns about outdoor activity timing",
            ],
            max_turns=8,
        ),
        ConversationScenario(
            scenario_id="flight_booking",
            description="User wants help finding a flight",
            starting_prompt="I need to fly from London to New York next Friday.",
            goals=[
                "User receives flight options",
                "User gets pricing information",
            ],
            max_turns=10,
        ),
    ]

How it works:

1. The CLI detects get_scenarios() (or a SCENARIOS constant) and switches to simulator mode for that file.
2. Each ConversationScenario becomes one eval case.
3. The simulator drives up to max_turns turns, generating contextual user messages after each agent response.
4. SimulationGoalsCriterion uses an LLM judge to score how many goals were achieved across the full conversation.
5. Pass/fail and the report are produced the same way as regular eval cases.

ConversationScenario fields:

SIMULATOR_CONFIG fields:

Config priority

When multiple sources configure the same setting, this priority applies (highest first):

1. CLI flags          (--parallel, --max-concurrency, --threshold, --output)
2. 10xgraph.json     "evaluation" section
3. Per-file config    get_eval_config() / EVAL_CONFIG  (inside each eval file)
4. confeval.py        get_eval_config() / EVAL_CONFIG  (project-root global fallback)
5. Built-in defaults  (all criteria at 0.6 threshold)

Common scenarios

Fast local check, single file, open report:

10xgraph eval evals/weather_agents_eval.py --open

Parallel run with 8 concurrent cases:

10xgraph eval --parallel --max-concurrency 8

Strict CI gate at 80% pass rate:

{
  "evaluation": {
    "threshold": 0.8,
    "parallel": true,
    "max_concurrency": 8
  }
}

10xgraph eval

Run only a regression suite in a subdirectory:

10xgraph eval evals/regression/ --output reports/regression

Mix regular evals and user simulator in the same run:

evals/
  weather_agents_eval.py     ← get_eval_set() protocol
  user_simulator_eval.py     ← get_scenarios() protocol

10xgraph eval --parallel

Both files are discovered, cases and scenarios are collected into the same flat pool, and results appear in a single merged report.

Common issues

“Eval directory ‘evals/’ not found”

• Create an evals/ directory or pass a path explicitly: 10xgraph eval path/to/evals
• Run 10xgraph init and choose the Production setup at the prompt to scaffold the standard project layout, which includes evals/.

“No eval cases found”

• Eval files must expose get_eval_set(), get_scenarios(), SCENARIOS, or functions annotated -> EvalSet.
• File must match *_eval.py or eval_*.py. Rename it or pass it explicitly.

File skipped with warning

• The file does not expose any recognised entry point. Add get_eval_set() or get_scenarios().

Exit code 1 even when all cases pass

• Check if a threshold is set in 10xgraph.json or passed via --threshold. The exit code is 1 when the pass rate is below threshold or when any case fails.

Simulator scenarios always fail

• Ensure the agent is reachable: either expose app in the eval file or set "agent" in 10xgraph.json.
• Check that goals are specific enough for the LLM judge to verify. Vague goals like “have a conversation” will not score well.
• Increase max_turns if the agent needs more exchanges to satisfy all goals.