Trace semantics for tool-using agents
A trace model for agentic workflows that makes tool calls, retries, policy gates, context mutations, and failure attribution visible enough for debugging and assurance.
Which trace fields are sufficient to attribute tool-using agent failures while avoiding unnecessary capture of sensitive prompt, tool, and user data.
- Span taxonomy
- Failure attribution model
- Sensitive-field trace policy
- Trace waterfall examples
Instrument a toy tool-using agent with the proposed span taxonomy, capture a full run transcript, and compare failure attribution quality against conventional request logs. The experiment should verify which span types are necessary, which fields can be safely redacted, and whether reviewers can identify the causal step without raw payload access.
Entropella Labs. Trace semantics for tool-using agents. Research note ENT-RN-002. Updated 5 Jul 2026.
Agent reliability improves when traces encode decisions, tools, policies, retries, state changes, and validation results as first-class events rather than treating the final model response as the only observable artifact.
Final responses erase the execution path
A conventional request log can tell a team that an AI system returned an answer. It usually cannot explain why the agent chose a tool, whether the retrieved context was stale, whether a retry changed the state, whether a policy gate was bypassed, or whether the final response depended on a partial failure.
The right unit of observation is the agent run: a directed sequence of model calls, tool invocations, state reads/writes, validation checks, policy decisions, and user-visible outputs. Each step needs enough metadata to support failure attribution without over-collecting sensitive data.
The minimum useful span taxonomy
| Span type | Purpose | Critical fields |
|---|---|---|
| agent.run | Root run for one user-facing workflow | run_id, user_intent, policy_profile, outcome |
| model.generate | Model call used for planning, tool choice, or response | model, prompt_hash, token_count, temperature, output_hash |
| tool.call | External function/API/file/browser/database call | tool_name, input_schema_version, permission_scope, status |
| retrieval.query | Search/vector/database retrieval | index_id, query_hash, top_k, context_ids, retrieval_score |
| policy.check | Guardrail, permission, or boundary evaluation | policy_id, decision, reason_code, override_user |
| state.write | Mutation to memory, database, ticket, or external system | target, diff_hash, reversible, confirmation_required |
| validator.run | Post-step check against schema, policy, or expected state | validator_id, pass, failure_type, remediation |
Instrumenting agent steps without leaking raw data
type SafeTraceEvent = {
runId: string;
spanType: "model.generate" | "tool.call" | "policy.check" | "state.write";
name: string;
startedAt: string;
endedAt?: string;
attributes: Record<string, string | number | boolean>;
sensitiveFields?: string[];
};
function hashForTrace(value: unknown) {
return createStableHash(JSON.stringify(value));
}
export function recordToolCall(runId: string, toolName: string, input: unknown) {
return emitTrace({
runId,
spanType: "tool.call",
name: toolName,
startedAt: new Date().toISOString(),
attributes: {
input_hash: hashForTrace(input),
input_schema_version: "2026-07",
permission_scope: "read:case-record",
},
sensitiveFields: ["customer_name", "email", "medical_notes"],
});
}Hashing and field classification do not eliminate privacy risk, but they reduce unnecessary collection. The point is to preserve diagnostic structure while avoiding raw sensitive payloads in routine traces.
Failures should be assigned to the causal step
- planning_error: the agent selected the wrong strategy before using a tool.
- retrieval_error: the agent relied on irrelevant, stale, missing, or conflicting context.
- tool_selection_error: the agent chose the wrong tool for the task.
- tool_input_error: the tool was right, but the parameters were invalid.
- policy_error: the workflow crossed or misapplied a control boundary.
- state_error: the agent wrote, updated, or deleted the wrong state.
- recovery_error: the agent detected an issue but failed to recover safely.
Related research
View allA measurement model for AI deployment readiness
How should teams measure whether an AI system is ready for controlled production deployment?
Reliability beyond pass rate
Which metrics better capture agent reliability than a single task success rate?
Stateful agent evaluation with transactional workflows
How should agents be evaluated when success depends on changing external state correctly?