Research library
ENT-RN-002Protocol draftedEvidence package

Trace semantics for tool-using agents

A trace model for agentic workflows that makes tool calls, retries, policy gates, context mutations, and failure attribution visible enough for debugging and assurance.

agent tracesOpenTelemetrytool callsobservability
Technical uncertainty

Which trace fields are sufficient to attribute tool-using agent failures while avoiding unnecessary capture of sensitive prompt, tool, and user data.

Expected evidence
  • Span taxonomy
  • Failure attribution model
  • Sensitive-field trace policy
  • Trace waterfall examples
Next experiment

Instrument a toy tool-using agent with the proposed span taxonomy, capture a full run transcript, and compare failure attribution quality against conventional request logs. The experiment should verify which span types are necessary, which fields can be safely redacted, and whether reviewers can identify the causal step without raw payload access.

Sources
Suggested citation

Entropella Labs. Trace semantics for tool-using agents. Research note ENT-RN-002. Updated 5 Jul 2026.

Hypothesis

Agent reliability improves when traces encode decisions, tools, policies, retries, state changes, and validation results as first-class events rather than treating the final model response as the only observable artifact.

Observation gap

Final responses erase the execution path

A conventional request log can tell a team that an AI system returned an answer. It usually cannot explain why the agent chose a tool, whether the retrieved context was stale, whether a retry changed the state, whether a policy gate was bypassed, or whether the final response depended on a partial failure.

The right unit of observation is the agent run: a directed sequence of model calls, tool invocations, state reads/writes, validation checks, policy decisions, and user-visible outputs. Each step needs enough metadata to support failure attribution without over-collecting sensitive data.

Agent trace waterfall
01
user request
02
planner model call
03
policy gate
04
retrieval span
05
tool call
06
validator
07
retry branch
08
human escalation
09
final response
Semantic model

The minimum useful span taxonomy

Span typePurposeCritical fields
agent.runRoot run for one user-facing workflowrun_id, user_intent, policy_profile, outcome
model.generateModel call used for planning, tool choice, or responsemodel, prompt_hash, token_count, temperature, output_hash
tool.callExternal function/API/file/browser/database calltool_name, input_schema_version, permission_scope, status
retrieval.querySearch/vector/database retrievalindex_id, query_hash, top_k, context_ids, retrieval_score
policy.checkGuardrail, permission, or boundary evaluationpolicy_id, decision, reason_code, override_user
state.writeMutation to memory, database, ticket, or external systemtarget, diff_hash, reversible, confirmation_required
validator.runPost-step check against schema, policy, or expected statevalidator_id, pass, failure_type, remediation
Trace completeness should be measured at the workflow-step level. A trace that captures every model token but misses the state mutation is not complete for assurance.
Implementation sketch

Instrumenting agent steps without leaking raw data

agent-trace.tsts
type SafeTraceEvent = {
  runId: string;
  spanType: "model.generate" | "tool.call" | "policy.check" | "state.write";
  name: string;
  startedAt: string;
  endedAt?: string;
  attributes: Record<string, string | number | boolean>;
  sensitiveFields?: string[];
};

function hashForTrace(value: unknown) {
  return createStableHash(JSON.stringify(value));
}

export function recordToolCall(runId: string, toolName: string, input: unknown) {
  return emitTrace({
    runId,
    spanType: "tool.call",
    name: toolName,
    startedAt: new Date().toISOString(),
    attributes: {
      input_hash: hashForTrace(input),
      input_schema_version: "2026-07",
      permission_scope: "read:case-record",
    },
    sensitiveFields: ["customer_name", "email", "medical_notes"],
  });
}

Hashing and field classification do not eliminate privacy risk, but they reduce unnecessary collection. The point is to preserve diagnostic structure while avoiding raw sensitive payloads in routine traces.

Failure attribution

Failures should be assigned to the causal step

Recommended failure labels
  • planning_error: the agent selected the wrong strategy before using a tool.
  • retrieval_error: the agent relied on irrelevant, stale, missing, or conflicting context.
  • tool_selection_error: the agent chose the wrong tool for the task.
  • tool_input_error: the tool was right, but the parameters were invalid.
  • policy_error: the workflow crossed or misapplied a control boundary.
  • state_error: the agent wrote, updated, or deleted the wrong state.
  • recovery_error: the agent detected an issue but failed to recover safely.
References

Related research