ENT-RN-011Protocol draftedEvidence package

Evidence graphs for reproducible AI assurance

A graph model for AI assurance evidence that preserves lineage from research question through experiment execution, artifact capture, failure analysis, and deployment decision.

evidence graphreproducibilitylineageassurance

Technical uncertainty

Whether a lightweight evidence graph can preserve enough lineage to answer review questions about claims, protocols, runs, artifacts, failures, limitations, and decisions.

Expected evidence

Graph schema
Artifact lineage model
Evidence query examples
Decision record template

Next experiment

Encode three synthetic experiment records as graph nodes and test whether stale evidence, open blockers, claim lineage, and artifact coverage can be queried directly. The result should prove that a reviewer can move from a public claim to the underlying run, metric, limitation, and decision without reconstructing context manually.

Sources

References

Hypothesis

Evidence graphs improve reproducibility and review quality by linking claims, protocols, datasets, model versions, prompts, traces, metrics, failures, conclusions, and limitations in a single inspectable structure.

Evidence problem

Research notes should preserve lineage, not just conclusions

AI assurance work often produces scattered artifacts: notebooks, screenshots, prompt files, traces, tickets, benchmark reports, model cards, and meeting decisions. When those artifacts are not linked, reviewers cannot tell which conclusion depends on which evidence or whether a deployment claim is stale.

An evidence graph makes the lineage explicit. A claim points to hypotheses, hypotheses point to experiments, experiments point to datasets and code, runs point to traces and metrics, and conclusions point to limitations and decisions.

Assurance evidence graph

claim

hypothesis

protocol

dataset

model version

prompt version

run

trace

metric

failure

limitation

decision

Schema

A minimal graph should be boring and strict

evidence-node.schema.jsonjson

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "EvidenceNode",
  "type": "object",
  "required": ["id", "type", "title", "created_at", "links"],
  "properties": {
    "id": { "type": "string" },
    "type": {
      "enum": [
        "claim",
        "hypothesis",
        "protocol",
        "dataset",
        "model_version",
        "prompt_version",
        "run",
        "trace",
        "metric",
        "failure",
        "limitation",
        "decision"
      ]
    },
    "title": { "type": "string" },
    "created_at": { "type": "string", "format": "date-time" },
    "owner": { "type": "string" },
    "links": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["rel", "target"],
        "properties": {
          "rel": { "type": "string" },
          "target": { "type": "string" }
        }
      }
    }
  }
}

Node	Purpose	Example link
claim	Statement under review	supported_by hypothesis
protocol	How evidence was produced	uses dataset
run	Execution instance	produced trace
metric	Quantified result	derived_from run
failure	Observed defect	occurred_in trace
decision	Human action	based_on claim

Queries

The graph should answer review questions quickly

evidence queriessql

-- Which deployment claims depend on stale model versions?
SELECT claim.id, claim.title, model.version, model.created_at
FROM claim
JOIN edge e1 ON e1.source = claim.id AND e1.rel = 'supported_by'
JOIN run ON run.id = e1.target
JOIN edge e2 ON e2.source = run.id AND e2.rel = 'used_model'
JOIN model_version model ON model.id = e2.target
WHERE model.created_at < NOW() - INTERVAL '45 days';

-- Which high-severity failures remain linked to an open decision?
SELECT failure.id, failure.title, decision.id
FROM failure
JOIN edge e ON e.source = decision.id AND e.target = failure.id
WHERE failure.severity >= 4 AND decision.status != 'closed';

Lineage completeness

0-1

Fraction of claims linked to protocol, run, metric, limitation, and decision.

Stale evidence age

days

Age of oldest artifact supporting an active deployment claim.

Open blocker count

integer

Unresolved high-severity failures linked to a candidate release.

Reader value

Evidence graphs make uncertainty reviewable

The goal is not to make research look more certain than it is. The goal is the opposite: make uncertainty precise, searchable, and reviewable. A good evidence graph helps a researcher ask what was tested, what was not tested, what changed, and which conclusion should no longer be trusted.

The best assurance artifact is one that lets a skeptical reviewer reproduce the chain from claim to evidence to limitation without asking the original author to reconstruct context from memory.

Implementation path

Start with static records, then graduate to a graph store

Begin with typed JSON/MD records for notes, protocols, runs, and artifacts.
Assign stable IDs to every claim, experiment, dataset, trace, and decision.
Render human-readable research notes from the same records used for machine queries.
Export to JSON-LD or a graph database once volume and query complexity justify it.
Never let the graph become append-only theater; stale evidence needs explicit invalidation.

References

W3C PROV-DM W3C PROV-Overview PROV-AGENT: Unified Provenance for Tracking AI Agent Interactions in Agentic Workflows

Related research

View all

ENT-RN-001