Eval Presets & Configuration

EvalConfig defines which criteria run and at what thresholds. EvalPresets provides ready-made configs so you can get meaningful results without writing config from scratch.

Built-in presets

All presets are factory class methods on EvalPresets. Every preset that calls an LLM accepts an optional judge_model parameter (default: "gemini-2.5-flash").

quick_check — Fastest, no LLM

Uses ROUGE-1 token overlap only. No API calls, instant feedback. Good for a smoke test during active development.

from tenxgraph.qa.evaluation import EvalPresets

config = EvalPresets.quick_check()

Criteria included:

• rouge_match (threshold 0.5)

tool_usage — Verify tool selection

Checks that the agent called the right tools with the right arguments. No LLM required.

config = EvalPresets.tool_usage(
    threshold=1.0,
    strict=True,      # EXACT match; set False for IN_ORDER
    check_args=True,
)

Criteria included:

• tool_name_match_score — tool names match
• tool_trajectory_avg_score — tool sequence matches

When to use: any agent that must call specific tools (weather, search, database lookups). This is the most common first-pass eval.

response_quality — Semantic accuracy

Uses an LLM to evaluate whether the agent’s response is correct and relevant.

config = EvalPresets.response_quality(
    threshold=0.7,
    use_llm_judge=True,
    judge_model="gemini-2.5-flash",
)

Criteria included:

• response_match_score — semantic match with expected response
• final_response_match_v2 — the llm_judge slot, added when use_llm_judge=True (single sample)

When to use: Q&A agents, FAQ bots, or any agent where response accuracy matters but exact wording varies.

Both criteria run the same semantic check; the second gives you a corroborating score under a separate name.

conversation_flow — Multi-turn quality

Validates both response quality and tool sequencing in multi-turn conversations.

config = EvalPresets.conversation_flow(
    threshold=0.8,
    judge_model="gemini-2.5-flash",
)

Criteria included:

• response_match_score — semantic match with the expected response
• tool_trajectory_avg_score — tool sequence with IN_ORDER matching

When to use: agents with multi-turn dialogue where the conversation follows a predictable flow.

safety_check — Safety and hallucination

Focused on what the agent outputs rather than whether it answers correctly.

config = EvalPresets.safety_check(
    threshold=0.8,
    judge_model="gemini-2.5-flash",
)

Criteria included:

• hallucinations_v1 — groundedness check
• safety_v1 — harmful content, hate speech, privacy, misinformation, manipulation

When to use: customer-facing agents, regulated industries, or any agent you are about to put in production.

comprehensive — All criteria

Runs every available criterion. Use this before a major release or for a thorough regression check.

config = EvalPresets.comprehensive(
    threshold=0.8,
    use_llm_judge=True,
    judge_model="gemini-2.5-flash",
)

Criteria included (no-LLM):

• tool_name_match_score
• tool_trajectory_avg_score
• rouge_match

Criteria included (LLM-judge, when use_llm_judge=True):

• final_response_match_v2 (the llm_judge slot)
• factual_accuracy_v1
• hallucinations_v1
• safety_v1

Note that comprehensive uses rouge_match rather than response_match_score for response comparison.

Note: contains_keywords is not included in comprehensive because keywords are domain-specific. Add it manually if needed (see Criteria — contains_keywords).

custom — Build from parameters

EvalPresets.custom() lets you enable exactly the criteria you need by passing threshold values. Any criterion whose threshold is None is excluded.

from tenxgraph.qa.evaluation import EvalPresets, MatchType

config = EvalPresets.custom(
    response_threshold=0.7,
    tool_threshold=1.0,
    llm_judge_threshold=None,           # excluded
    hallucination_threshold=0.8,
    safety_threshold=0.8,
    factual_accuracy_threshold=None,    # excluded
    tool_match_type=MatchType.IN_ORDER,
    check_tool_args=True,
    judge_model="gpt-4o",
)

combine — Merge presets

EvalPresets.combine() merges multiple configs. Later configs overwrite earlier ones when keys conflict.

config = EvalPresets.combine(
    EvalPresets.tool_usage(threshold=1.0),
    EvalPresets.safety_check(threshold=0.8),
)

EvalConfig built-in presets

EvalConfig also ships three class-method presets:

from tenxgraph.qa.evaluation import EvalConfig

config = EvalConfig.default()   # EXACT trajectory + semantic response match
config = EvalConfig.strict()    # EXACT trajectory with args + high-threshold judges
config = EvalConfig.relaxed()   # IN_ORDER trajectory + lower response threshold

All three use response_match, which is LLM-based. None of them use ROUGE — for a no-LLM config use EvalPresets.quick_check().

Save and load config

# Save to JSON
config.to_file("eval_config.json")

# Load from JSON
config = EvalConfig.from_file("eval_config.json")

Building a custom EvalConfig from scratch

EvalConfig.criteria is a typed CriteriaConfig model with one named slot per criterion. It forbids unknown fields, so you cannot invent your own keys such as "tone_check" — each criterion goes in the slot that belongs to it.

from tenxgraph.qa.evaluation import (
    CriteriaConfig,
    CriterionConfig,
    EvalConfig,
    MatchType,
    Rubric,
)

config = EvalConfig(
    criteria=CriteriaConfig(
        # No-LLM: tool correctness
        trajectory=CriterionConfig.trajectory(
            threshold=1.0,
            match_type=MatchType.EXACT,
            check_args=True,
        ),

        # No-LLM: response overlap
        rouge_match=CriterionConfig.rouge_match(threshold=0.5),

        # LLM: semantic accuracy
        response_match=CriterionConfig.response_match(
            threshold=0.8,
            judge_model="gemini-2.5-flash",
            num_samples=3,
        ),

        # LLM: hallucination
        hallucination=CriterionConfig.hallucination(threshold=0.9),

        # LLM: domain-specific rubrics — all rubrics share the one slot
        rubric_based=CriterionConfig.rubric_based(
            rubrics=[
                Rubric(
                    rubric_id="professional_tone",
                    content=(
                        "The response must use professional, formal language. "
                        "Avoid colloquialisms and slang."
                    ),
                ),
            ],
            threshold=0.8,
        ),

        # Keyword check
        contains_keywords=CriterionConfig.contains_keywords(
            keywords=["consult a professional", "not financial advice"],
            threshold=1.0,
        ),
    ),
    parallel=True,
    max_concurrency=4,
    timeout=120.0,
)

Available slots: tool_name_match, trajectory, node_order, response_match, rouge_match, contains_keywords, llm_judge, rubric_based, factual_accuracy, hallucination, safety, simulation_goals. Because there is exactly one rubric_based slot, multiple named rubrics all go into the same rubrics list rather than into separate criteria.

Add rubrics to an existing config

from tenxgraph.qa.evaluation import Rubric

config = config.with_rubrics([
    Rubric(rubric_id="concise", content="Response must be under 50 words.", weight=1.0),
])

Choosing criteria for your use case

Start fast (no-LLM criteria) and add LLM-judge criteria incrementally as your eval suite matures.

Next steps

• Reports — how scores are presented in HTML, JSON, and JUnit XML
• Criteria reference — full details on each criterion
• Eval sets — building test cases