Stateful agent evaluation with transactional workflows
A τ-bench-inspired protocol for evaluating agents in realistic user-tool environments where the final database state is the source of truth.
Whether final environment state and policy-violation scoring better predict real workflow safety than answer-only grading for multi-turn tool-using agents.
- State transition model
- User simulator spec
- Goal-state scorer
- Policy violation report
Build a small transactional simulator with a mutable record store, policy guide, user simulator, and goal-state scorer. Use it to test whether final state accuracy, policy violations, and recovery behavior stay legible when the workflow requires multiple tool calls, clarifications, and corrective actions.
Transactional agent evaluation is more deployment-relevant than answer-only scoring because the true outcome is the final state of the environment after user turns, tool calls, policy constraints, and recovery attempts.
Agents should be judged by the world they leave behind
A transactional agent can produce a polished answer while leaving a corrupted order, an incorrect refund, a duplicated booking, or an unresolved support case. Answer-only scoring misses these failures because the user-visible text is not the operational outcome.
The evaluation should model a workflow environment with state, domain policies, API tools, and a user simulator. The scorer compares the final state after the conversation with an annotated goal state, while also measuring whether the agent violated policies on the way there.
A minimum viable transactional benchmark
| Component | Requirement | Reason |
|---|---|---|
| User simulator | Paraphrases, withholds, corrects, and challenges information | Prevents overfitting to one scripted interaction |
| Tool layer | Typed APIs with realistic errors and permission boundaries | Exposes tool-use reliability rather than pure chat quality |
| Policy guide | Domain-specific rules and exceptions | Tests whether the agent follows operational constraints |
| State database | Mutable records with a known initial and goal state | Allows objective scoring after the run |
| Transcript and trace | Full decision path with redaction of sensitive fields | Enables failure attribution and auditability |
def score_goal_state(initial_state, final_state, expected_state, policy_events):
field_scores = {}
for path, expected in expected_state.items():
observed = get_path(final_state, path)
field_scores[path] = observed == expected
blocking_policy_failures = [
event for event in policy_events
if event["severity"] >= 4 and event["decision"] == "violated"
]
return {
"goal_state_match": all(field_scores.values()),
"field_scores": field_scores,
"policy_safe": len(blocking_policy_failures) == 0,
"blocking_policy_failures": blocking_policy_failures,
}What makes transactional evals misleading
- Toy APIs make agents look more reliable than production tools with partial failures and ambiguous errors.
- A passive user simulator under-tests clarification, refusal, and recovery behavior.
- Goal-state scoring can miss policy violations if only the final database state is inspected.
- Single-attempt results hide run-to-run instability.
- Benchmarks become stale if task templates leak into prompt engineering loops.
The useful artifact is a living environment
The highest-value deliverable is not a static leaderboard. It is a versioned environment where policies, tools, task distributions, and state schemas can evolve as real deployments evolve. That makes regressions visible and prevents evaluation from becoming a one-time gate.
Related research
View allReliability beyond pass rate
Which metrics better capture agent reliability than a single task success rate?
A measurement model for AI deployment readiness
How should teams measure whether an AI system is ready for controlled production deployment?
Trace semantics for tool-using agents
What trace semantics are needed to explain failures in tool-using AI agents?