Research notes
A public research library for AI assurance infrastructure. Each note states the question, hypothesis, expected evidence, and current maturity so planned work is never presented as finished.
11 of 11 notes
A measurement model for AI deployment readiness
How should teams measure whether an AI system is ready for controlled production deployment?
Trace semantics for tool-using agents
What trace semantics are needed to explain failures in tool-using AI agents?
Reliability beyond pass rate
Which metrics better capture agent reliability than a single task success rate?
Stateful agent evaluation with transactional workflows
How should agents be evaluated when success depends on changing external state correctly?
Privacy boundary testing for hybrid local/cloud AI
How can teams verify that sensitive data stays inside intended local, controlled, or cloud execution boundaries?
Prompt injection and tool misuse in agentic workflows
How should agentic systems be tested against prompt injection, tool misuse, and excessive agency?
RAG evaluation under domain shift
How should retrieval-augmented generation be evaluated when source documents, user questions, and domain language shift over time?
LLM-as-judge reliability, calibration, and bias
When can LLM judges be trusted as scalable evaluators, and when do they need human calibration?
Benchmark contamination and evaluation leakage
How can AI evaluations avoid overstating performance because the model, prompt, or team has already seen the test?
Cost-latency-reliability frontiers for compound AI systems
How should teams reason about tradeoffs between model quality, retries, routing, cost, and latency in compound AI systems?
Evidence graphs for reproducible AI assurance
Can AI assurance evidence be represented as a queryable graph from hypothesis to experiment to result to decision?