ENT-RN-009Protocol draftedProtocol drafted

Benchmark contamination and evaluation leakage

A leakage-control protocol for AI evaluations, covering dataset contamination, prompt overfitting, benchmark saturation, and held-out task governance.

benchmark leakagecontaminationholdout setsevaluation

Technical uncertainty

Whether exposure tracking, near-duplicate scanning, and separate regression versus holdout sets can reduce benchmark leakage enough for credible deployment decisions.

Expected evidence

Leakage threat model
Holdout governance protocol
Dataset fingerprinting sketch
Prompt tuning audit

Next experiment

Create a toy benchmark with deliberately leaked and clean holdout items, then measure performance delta and near-duplicate detection quality. The experiment should also record exposure provenance, tuning iterations, and which items remain safe to use as a private decision set.

Sources

References

Hypothesis

Evaluation credibility depends on leakage controls across datasets, prompts, model training data, retrieval corpora, tuning loops, and analyst behavior because contamination can make brittle systems appear robust.

Evaluation risk

Leakage can turn evaluation into rehearsal

A model can perform well because it generalizes, because it memorized benchmark material, because developers tuned prompts against the test, or because the retrieval corpus contains answer keys. These cases produce similar headline scores but radically different deployment meaning.

Contamination pathways

public benchmark

training corpus

prompt tuning loop

retrieval index

developer memory

evaluation report

deployment claim

Controls

Separate public regression tests from private decision tests

Test set	Purpose	Access policy
Public benchmark	Comparable external signal	Open, but never sufficient for deployment readiness
Regression suite	Detect known failures after changes	Visible to developers; expected to be optimized
Private holdout	Estimate generalization on unseen tasks	Limited access; no prompt tuning on failures
Fresh incident-derived set	Test recent failure modes	Rotating set with strict lineage tracking

near-duplicate scan sketchpython

def flag_potential_leakage(eval_items, known_corpus, embed):
    corpus_vectors = [(doc["id"], embed(doc["text"])) for doc in known_corpus]
    flags = []

    for item in eval_items:
        item_vec = embed(item["prompt"] + "\n" + item.get("answer", ""))
        nearest = max(corpus_vectors, key=lambda row: cosine(item_vec, row[1]))
        if cosine(item_vec, nearest[1]) > 0.92:
            flags.append({
                "eval_id": item["id"],
                "nearest_corpus_id": nearest[0],
                "reason": "semantic_near_duplicate",
            })
    return flags

Operational discipline

Track exposure as carefully as score

Record which team members can view private holdout samples.
Separate failure analysis samples from final decision samples.
Limit prompt iterations against any fixed evaluation set.
Rotate fresh tasks from realistic workflow changes.
Report performance on public, regression, and holdout sets separately.
Keep provenance for generated or synthetic evaluation items.

Interpretation

A lower clean score may be more valuable than a higher leaked score

For deployment decisions, credibility matters more than optics. A lower score on a well-controlled holdout can be more informative than a high score on a saturated public benchmark. The library should make that distinction explicit.

References

Benchmark Data Contamination of Large Language Models: A Survey An Open Source Data Contamination Report for Large Language Models HELM: Holistic Evaluation of Language Models

Related research

View all

ENT-RN-008