ENT-RN-003Protocol draftedEvidence package

Reliability beyond pass rate

A reliability framework for tool-using agents that treats variance, recovery, failure impact, and state correctness as deployment-critical signals.

reliabilitypass@kvariancefailure severity

Technical uncertainty

Whether repeated-run variance, state correctness, recovery quality, and severity-weighted failure scoring produce a more useful deployment signal than binary task pass rate alone.

Expected evidence

Repeated-run protocol
Failure severity scale
Confidence interval report
Recovery-quality rubric

Next experiment

Run a synthetic task suite across repeated agent trials and compare pass rate, pass@k, state match, severity-weighted failure rate, and recovery success. The result should include confidence intervals, a failure-severity histogram, and examples of the worst tail cases so reliability can be reviewed as a distribution instead of a single score.

Sources

References

Artifacts

Synthetic reliability run summary

Suggested citation

Entropella Labs. Reliability beyond pass rate. Research note ENT-RN-003. Updated 2 Jul 2026.

Hypothesis

Agent reliability requires repeated trials, variance estimates, severity-weighted failures, recovery scoring, and state-based correctness because a binary pass rate can hide brittle behavior in long-horizon workflows.

Measurement flaw

A single success rate is too lossy for agents

Binary success rate is useful, but it is not sufficient. In a long-horizon agent workflow, a run can succeed for fragile reasons, fail safely, fail dangerously, recover after a near miss, or leave hidden state damage despite producing a plausible final answer.

The evaluation target should be the distribution of behavior across repeated runs. A system that succeeds 70% of the time with low-impact failures is different from a system that succeeds 70% of the time but occasionally deletes records, leaks data, or fabricates irreversible actions.

Metric stack

Measure reliability as a vector, not a scalar

Metric	What it reveals	Failure mode it catches
pass@1	Single-run completion probability	Basic task inability
pass@k	Whether success appears across repeated attempts	Instability and brittle planning
state match rate	Whether the final system state is correct	Plausible answer with wrong database state
severity-weighted failure rate	Impact-adjusted failure burden	Rare but unacceptable failures
recovery success	Ability to detect and correct mistakes	Unrecoverable intermediate errors
trace-localized defect rate	Where failures originate	Unclear ownership across model/tool/policy layers

severity-weighted failure ratepython

from statistics import mean

SEVERITY = {
    "cosmetic": 1,
    "minor_wrong_answer": 2,
    "workflow_blocked": 3,
    "incorrect_state_write": 4,
    "privacy_or_security_breach": 5,
}

def severity_weighted_failure_rate(runs):
    weighted = []
    for run in runs:
        if run["passed"]:
            weighted.append(0)
        else:
            weighted.append(SEVERITY[run["failure_type"]] / 5)
    return mean(weighted)

def reliability_vector(runs):
    return {
        "pass_at_1": mean(int(run["passed"]) for run in runs),
        "state_match": mean(int(run["state_matches_goal"]) for run in runs),
        "recovery_success": mean(int(run["recovered_safely"]) for run in runs if run["had_error"]),
        "severity_weighted_failure_rate": severity_weighted_failure_rate(runs),
    }

Protocol

Run design matters as much as scoring

Use repeated runs for each task with controlled variation in seeds, paraphrases, or user simulator behavior.
Score the final environment state, not only the final message.
Record every intermediate tool call and state mutation.
Use confidence intervals when comparing models or prompts.
Separate recoverable failures from silent dangerous failures.
Retain negative examples; they are often more valuable than aggregate metrics.

Reliability distribution

For deployment review, the tail of the distribution matters more than the mean.

task set

30 repeated runs

state scorer

failure labeler

severity weighting

confidence interval

deployment decision

Interpretation

The deployment question is about unacceptable tails

A serious reliability note should not celebrate a high average while ignoring tail risk. For AI systems that can invoke tools, write state, or influence decisions, the key question is whether the worst observed failures are acceptable, containable, and detectable.

References

Evaluating Large Language Models Trained on Code HELM: Holistic Evaluation of Language Models tau-bench: Tool-Agent-User interaction benchmark

Related research

View all

ENT-RN-004