ENT-RN-010Experiment activeExperiment active

Cost-latency-reliability frontiers for compound AI systems

A profiler framework for measuring the economic and operational frontier of multi-step AI workflows across model routes, retries, caching, and validation.

costlatencycompound AImodel routing

Technical uncertainty

Whether step-level profiling can identify non-dominated workflow configurations across reliability, p95 latency, and cost per successful run.

Expected evidence

Cost waterfall
Latency distribution
Pareto frontier chart
Routing policy comparison

Next experiment

Run the same synthetic workflow through four routing policies and compute the cost-latency-reliability frontier. Include step-level token counts, queue time, retry overhead, and a Pareto table so the preferred route can be justified by evidence rather than intuition.

Sources

References

View all references

Hypothesis

Compound AI systems should be optimized on Pareto frontiers rather than single metrics because retries, tool calls, model routing, cache strategy, and validation can improve reliability while creating nonlinear cost and latency overhead.

Systems view

The expensive part is often the workflow, not the model call

A compound AI system may call several models, retrieve context, invoke tools, validate outputs, retry failures, escalate to humans, and write state. Cost and latency accumulate across the whole graph. Optimizing a single model call can be irrelevant if the retry policy or retrieval layer dominates runtime.

Cost waterfall

input classification

retrieval

planner model

tool call

validator model

retry branch

final model

human review queue

Frontier analysis

Optimize for dominated and non-dominated configurations

A configuration is dominated when another configuration is cheaper, faster, and at least as reliable. The useful design surface is the frontier of non-dominated options. Product teams can then choose a point on the frontier based on business constraints rather than arguing over a single global optimum.

Configuration	Reliability	p95 latency	Cost/run	Interpretation
Small model only	0.61	1.2s	$0.004	Cheap and fast, not reliable enough
Router + validator	0.78	2.8s	$0.014	Likely frontier candidate
Large model all steps	0.82	5.9s	$0.061	Higher quality, expensive
Router + retry + human gate	0.88	9.4s	$0.039	Best for high-impact cases

pareto-frontier.pypython

def is_dominated(candidate, configs):
    for other in configs:
        if other is candidate:
            continue
        cheaper = other["cost"] <= candidate["cost"]
        faster = other["p95_latency"] <= candidate["p95_latency"]
        better = other["reliability"] >= candidate["reliability"]
        strictly_better = (
            other["cost"] < candidate["cost"] or
            other["p95_latency"] < candidate["p95_latency"] or
            other["reliability"] > candidate["reliability"]
        )
        if cheaper and faster and better and strictly_better:
            return True
    return False

frontier = [config for config in configs if not is_dominated(config, configs)]

Profiler schema

Measure cost and latency at step level

Record tokens, model, provider, cache status, and route decision for every model call.
Separate latency waiting on model, tool, queue, network, and human review.
Compute cost per attempted run and cost per successful run.
Attribute retry overhead to the step that triggered the retry.
Report frontier movement after prompt, model, retriever, or policy changes.

Decision use

The frontier should drive routing policy

A mature system does not use one route for every case. Low-risk cases can use cheaper routes; high-uncertainty or high-impact cases can route to stronger models, additional validation, or human review. The frontier makes these tradeoffs explicit.

References

OpenTelemetry Generative AI semantic conventions Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systems HELM: efficiency as an evaluation dimension

Related research

View all

ENT-RN-001