10xGraphv0.9.2Search

    Evaluation criteria

    In shortConstructor signatures, defaults and scoring behaviour for every built-in evaluation criterion class in tenxgraph.qa.evaluation.criteria.

    • 9 min read
    • 10 sections
    • Updated
    • v0.9.2
    • Markdown

    When to use this

    Use this page when you need the class-level API: constructor arguments, the criterion name that appears in reports, and what the score means. It covers the criterion objects themselves.

    Two related pages cover the layers above:

    • Evaluation — AgentEvaluator, EvalSet, EvalCase, reports.
    • Criteria (concepts) — choosing thresholds and configuring criteria through CriterionConfig factory methods.

    You only need the classes directly when you are writing a custom criterion, composing criteria by hand, or calling one outside AgentEvaluator.

    Import paths

    Python
    from tenxgraph.qa.evaluation import (
    BaseCriterion,
    SyncCriterion,
    CompositeCriterion,
    WeightedCriterion,
    ExactMatchCriterion,
    ContainsKeywordsCriterion,
    ResponseMatchCriterion,
    RougeMatchCriterion,
    LLMJudgeCriterion,
    RubricBasedCriterion,
    FactualAccuracyCriterion,
    HallucinationCriterion,
    SafetyCriterion,
    TrajectoryMatchCriterion,
    NodeOrderMatchCriterion,
    ToolNameMatchCriterion,
    CriterionConfig,
    MatchType,
    Rubric,
)

    The classes also live under tenxgraph.qa.evaluation.criteria, together with CRITERIA_REGISTRY (the name → class mapping AgentEvaluator uses):

    Python
    from tenxgraph.qa.evaluation.criteria import CRITERIA_REGISTRY

    Which criterion do I want?

    If you want to check… Use Calls an LLM
    The response is byte-for-byte the expected string ExactMatchCriterion no
    Specific words appear in the response ContainsKeywordsCriterion no
    Word overlap with the expected answer RougeMatchCriterion no
    The answer means the same thing, however it is worded ResponseMatchCriterion yes
    Overall answer quality against the expected answer LLMJudgeCriterion yes
    Your own written grading rules RubricBasedCriterion yes
    Facts, numbers, dates and names are correct FactualAccuracyCriterion yes
    Claims are grounded in tool results / provided context HallucinationCriterion yes
    The response is harmless and policy-safe SafetyCriterion yes
    The right tools were called, in the right order TrajectoryMatchCriterion no
    The right tools were called, ignoring arguments ToolNameMatchCriterion no
    The graph routed through the right nodes NodeOrderMatchCriterion no
    Several criteria as one AND / OR gate CompositeCriterion depends
    Several criteria averaged by importance WeightedCriterion depends
    A custom rule of your own subclass BaseCriterion / SyncCriterion your choice

    Common interface

    Every criterion derives from BaseCriterion.

    Python
    from tenxgraph.qa.evaluation import BaseCriterion, CriterionConfig

criterion = SomeCriterion(config=CriterionConfig(threshold=0.9))
result = await criterion.evaluate(execution_result, eval_case)  # CriterionResult

    BaseCriterion.__init__

    Parameter Type Default Description
    config CriterionConfig | None None Threshold, judge model, match type and friends. None builds a default CriterionConfig().

    Members

    Member Kind Description
    name class attr str Criterion key written into CriterionResult.criterion and EvalSummary.criterion_stats.
    description class attr str Human-readable description.
    config attr CriterionConfig The active configuration.
    threshold property float Shortcut for config.threshold.
    is_enabled property bool Shortcut for config.enabled.
    evaluate async (actual: ExecutionResult, expected: EvalCase) -> CriterionResult Abstract. Scores one case.
    validate_config () -> list[str] Returns config error strings; empty list means valid.

    evaluate() is async on every criterion, including the deterministic ones — SyncCriterion simply awaits its own synchronous implementation.

    Configuration

    All criteria are configured with the same CriterionConfig model. Its fields and factory methods are documented in Criteria (concepts). Each criterion only reads the fields relevant to it, listed per class below.

    String and keyword criteria

    ExactMatchCriterion

    Python
    from tenxgraph.qa.evaluation import ExactMatchCriterion

    name = "exact_match". Compares actual.actual_response.strip() against the last non-empty expected_final_response in the case. Score is 1.0 or 0.0; there is no partial credit.

    Parameter Type Default Description
    config CriterionConfig | None None Only threshold and enabled are read.
    Python
    criterion = ExactMatchCriterion()
result = await criterion.evaluate(execution, case)
print(result.score, result.details["match"])

    CriteriaConfig has no exact_match field, so this criterion cannot be turned on through EvalConfig.criteria. Instantiate it directly, or look it up under the "exact_match" key in CRITERIA_REGISTRY.

    ContainsKeywordsCriterion

    Python
    from tenxgraph.qa.evaluation import ContainsKeywordsCriterion

    name = "contains_keywords". Case-insensitive substring search over the final response. Score is len(found) / len(keywords).

    Parameter Type Default Description
    keywords list[str] | None None Keywords to look for. Ignored when config.keywords is non-empty.
    config CriterionConfig | None None config.keywords takes precedence over the keywords argument.

    With no keywords from either source the criterion returns 1.0 and a {"note": "No keywords specified"} detail.

    Python
    from tenxgraph.qa.evaluation import ContainsKeywordsCriterion, CriterionConfig

criterion = ContainsKeywordsCriterion(
    config=CriterionConfig.contains_keywords(
        keywords=["refund", "7 days"],
        threshold=1.0,
    ),
)

    RougeMatchCriterion

    Python
    from tenxgraph.qa.evaluation import RougeMatchCriterion

    name = "rouge_match". ROUGE-1 F1 over lowercased, punctuation-stripped unigram sets. No LLM call. details carries precision, recall and f1, plus the first 500 characters of each response.

    Parameter Type Default Description
    config CriterionConfig | None None Only threshold and enabled are read.

    When the case defines no expected response, the score is 1.0 if the agent also produced nothing, otherwise 0.5.

    LLM-judged criteria

    These five share a base class, TemplatedLLMCriterion, which calls the judge model config.num_samples times (default 3), averages the score field of each JSON reply, and records the combined token usage on CriterionResult.token_usage. Samples that raise are logged and dropped; if every sample fails the criterion returns a failure result with error="All LLM samples failed".

    Shared constructor:

    Parameter Type Default Description
    config CriterionConfig | None None Reads threshold (0.8), judge_model ("gemini-2.5-flash"), num_samples (3), api_style ("responses"), and rubrics for RubricBasedCriterion.

    The judge provider is picked from judge_model; set GOOGLE_GENAI_USE_VERTEXAI=true to route Google models through Vertex AI. If no provider can be resolved, each sample degrades to score=0.5 with reasoning "No LLM provider available".

    ResponseMatchCriterion

    Python
    from tenxgraph.qa.evaluation import ResponseMatchCriterion

    name = "response_match_score". Subclass of LLMJudgeCriterion with the same prompt — semantic equivalence between actual and expected response. This is the recommended default over RougeMatchCriterion.

    Python
    from tenxgraph.qa.evaluation import CriterionConfig, ResponseMatchCriterion

criterion = ResponseMatchCriterion(
    config=CriterionConfig.response_match(threshold=0.8, num_samples=3),
)
result = await criterion.evaluate(execution, case)
print(result.score, result.details["reason"])

    LLMJudgeCriterion

    Python
    from tenxgraph.qa.evaluation import LLMJudgeCriterion

    name = "final_response_match_v2". Asks the judge whether the actual response is semantically equivalent to the expected one. Skips with score 1.0 when the case defines no expected final response, or when the agent produced no response.

    details: samples, individual_scores, reasonings, judge_model, reason.

    RubricBasedCriterion

    Python
    from tenxgraph.qa.evaluation import RubricBasedCriterion, Rubric

    name = "rubric_based_final_response_quality_v1". Renders every config.rubrics entry into the judge prompt as "- {rubric_id}: {content}". Returns 1.0 immediately when config.rubrics is empty.

    Python
    criterion = RubricBasedCriterion(
    config=CriterionConfig.rubric_based(
        rubrics=[
            Rubric(rubric_id="concise", content="Under 100 words.", weight=1.0),
            Rubric(rubric_id="empathetic", content="Acknowledges the complaint.", weight=0.5),
        ],
        threshold=0.8,
    ),
)

    details: rubrics (the ids), scores, reasonings.

    FactualAccuracyCriterion

    Python
    from tenxgraph.qa.evaluation import FactualAccuracyCriterion

    name = "factual_accuracy_v1". Reference material is the last expected final response, falling back to expected.metadata["reference_facts"].

    details: is_accurate (final_score >= threshold), samples, reasonings, errors (de-duplicated across samples).

    HallucinationCriterion

    Python
    from tenxgraph.qa.evaluation import HallucinationCriterion

    name = "hallucinations_v1". Builds the grounding context from every tool call result in the execution plus expected.metadata["context"], then asks whether the response is supported by it. 1.0 is fully grounded.

    details: is_grounded, samples, reasonings, hallucinations.

    SafetyCriterion

    Python
    from tenxgraph.qa.evaluation import SafetyCriterion

    name = "safety_v1". The judge returns one overall score plus per-category sub-scores for harmful_content, hate_speech, privacy, misinformation and manipulation. The criterion score is the average overall score across samples; the category values are averaged and reported as category_scores for triage.

    details: is_safe, samples, reasonings, issues, category_scores.

    Trajectory criteria

    All three are SyncCriterion subclasses — deterministic, no API calls. Expected values come from the Invocation entries of the case, concatenated across turns.

    TrajectoryMatchCriterion

    Python
    from tenxgraph.qa.evaluation import TrajectoryMatchCriterion

    name = "tool_trajectory_avg_score". Compares actual.tool_calls against every invocation.expected_tool_trajectory, using config.match_type and config.check_args.

    Config field Default Effect
    match_type MatchType.EXACT EXACT — same tools in the same positions. IN_ORDER — expected tools in order, extras allowed. ANY_ORDER — expected tools anywhere.
    check_args False When True, tool arguments must also match. An expected ToolCall with empty args accepts any arguments.
    threshold 0.8 Score is the fraction of expected tools matched.
    Python
    from tenxgraph.qa.evaluation import CriterionConfig, MatchType, TrajectoryMatchCriterion

criterion = TrajectoryMatchCriterion(
    config=CriterionConfig.trajectory(
        threshold=1.0,
        match_type=MatchType.IN_ORDER,
        check_args=True,
    ),
)

    With no expected tools, EXACT scores 1.0 only if the agent called nothing; IN_ORDER and ANY_ORDER score 1.0 unconditionally.

    NodeOrderMatchCriterion

    Python
    from tenxgraph.qa.evaluation import NodeOrderMatchCriterion

    name = "node_order_score". Same three match modes, applied to actual.node_visits versus invocation.expected_node_order. Reads match_type and threshold; check_args is not used.

    Python
    criterion = NodeOrderMatchCriterion(
    config=CriterionConfig.node_order(threshold=1.0, match_type=MatchType.IN_ORDER),
)

    ToolNameMatchCriterion

    Python
    from tenxgraph.qa.evaluation import ToolNameMatchCriterion

    name = "tool_name_match_score". Multiset comparison of tool names only, order and arguments ignored. Score is matched names over expected names. When nothing is expected, the score is 1.0 if the agent called no tools and 0.5 if it did.

    Composing criteria

    CompositeCriterion

    Python
    from tenxgraph.qa.evaluation import CompositeCriterion

    name = "composite_criterion". Runs every sub-criterion and combines the scores.

    Parameter Type Default Description
    criteria list[BaseCriterion] required Sub-criteria to run, in order.
    require_all bool True True (AND) takes the minimum score; False (OR) takes the maximum.
    config CriterionConfig | None None Supplies the threshold applied to the combined score.
    Python
    from tenxgraph.qa.evaluation import (
    CompositeCriterion,
    ContainsKeywordsCriterion,
    CriterionConfig,
    ToolNameMatchCriterion,
)

criterion = CompositeCriterion(
    criteria=[
        ToolNameMatchCriterion(),
        ContainsKeywordsCriterion(keywords=["refund"]),
    ],
    require_all=True,
    config=CriterionConfig(threshold=1.0),
)

    details contains sub_results (each sub-result serialised) and require_all.

    WeightedCriterion

    Python
    from tenxgraph.qa.evaluation import WeightedCriterion

    name = "weighted_criterion". Weighted average of the sub-scores; sum(score * weight) / sum(weight). Weights need not sum to 1.

    Parameter Type Default Description
    criteria_weights list[tuple[BaseCriterion, float]] required (criterion, weight) pairs.
    config CriterionConfig | None None Supplies the threshold applied to the weighted score.
    Python
    criterion = WeightedCriterion(
    criteria_weights=[
        (ToolNameMatchCriterion(), 0.3),
        (ResponseMatchCriterion(), 0.7),
    ],
    config=CriterionConfig(threshold=0.8),
)

    An empty criteria_weights (or total weight 0) scores 0.0.

    Writing your own criterion

    SyncCriterion

    Subclass SyncCriterion when your rule is pure computation. Implement evaluate_sync(); the inherited async evaluate() wraps it, so the criterion drops straight into AgentEvaluator.

    Python
    from tenxgraph.qa.evaluation import SyncCriterion

class MaxLengthCriterion(SyncCriterion):
    name = "max_length"
    description = "Response must stay under 500 characters"

    def evaluate_sync(self, actual, expected):
        length = len(actual.actual_response)
        return self._result(
            1.0 if length <= 500 else 0.0,
            {"length": length},
        )

    Subclass BaseCriterion instead when you need to await something inside the rule, and implement the async evaluate() directly.

    Two protected helpers from BaseCriterion build results for you:

    Helper Returns
    self._result(score, details=None, token_usage=None) Passing/failing result judged against self.threshold.
    self._failure(error) Error result with score=0.0, passed=False.

    To use a custom criterion through EvalConfig, register it before building the evaluator:

    Python
    from tenxgraph.qa.evaluation.criteria import CRITERIA_REGISTRY

CRITERIA_REGISTRY["max_length"] = MaxLengthCriterion

    CriteriaConfig forbids unknown fields, so a registry-only criterion still has to be attached to the evaluator by hand or mapped onto an existing field name.

    Common errors

    Error / symptom Cause Fix
    TypeError: Can't instantiate abstract class ... evaluate Subclassed BaseCriterion without an async evaluate(), or SyncCriterion without evaluate_sync(). Implement the abstract method.
    RuntimeWarning: coroutine 'evaluate' was never awaited evaluate() is async on every criterion, including the deterministic ones. await criterion.evaluate(...).
    Score is exactly 0.5 with reasoning "No LLM provider available" The judge model could not be resolved to a configured provider. Install the provider extra and set its API key; set GOOGLE_GENAI_USE_VERTEXAI=true for Vertex-only projects.
    CriterionResult.error == "All LLM samples failed" Every judge call raised. Check the tenxgraph.evaluation logger for the per-sample warnings.
    contains_keywords always scores 1.0 Neither keywords= nor config.keywords was set. Pass CriterionConfig.contains_keywords(keywords=[...]).
    ValueError: Unknown criterion field: 'exact_match' CriteriaConfig has no field for it. Instantiate ExactMatchCriterion directly.
    TrajectoryMatchCriterion fails on correct-looking runs MatchType.EXACT rejects extra tool calls. Switch to MatchType.IN_ORDER or ANY_ORDER.
    check_args=True never matches Expected ToolCall.args must equal the actual args exactly. Leave args={} to accept any arguments for that call.
    Last updated for v0.9.2Edit this page on GitHubReport an issue