ENT-RN-008Protocol draftedProtocol drafted

LLM-as-judge reliability, calibration, and bias

A calibration protocol for LLM-as-judge systems that treats judges as fallible measurement instruments rather than neutral truth sources.

LLM-as-judgecalibrationbiasevaluation

Technical uncertainty

Whether automated judges can produce stable, calibrated, and bias-aware scores for assurance tasks when compared against objective checks and small human review samples.

Expected evidence

Judge rubric
Bias test suite
Human agreement sample
Disagreement heatmap

Next experiment

Evaluate one rubric with swapped candidate order, controlled answer length, and a small expert-labelled sample to measure position and verbosity bias. The note should also include a disagreement table and calibration summary so the judge can be treated as a measured instrument rather than a black box.

Sources

References

Hypothesis

LLM judges are useful for scalable evaluation when their rubrics, biases, calibration, disagreement patterns, and failure modes are measured explicitly against human or objective references.

Measurement frame

A judge model is an instrument, not an oracle

LLM judges can reduce evaluation cost and make qualitative scoring repeatable, but they introduce their own measurement error. A judge can prefer longer answers, be sensitive to candidate order, over-reward confident style, miss domain-specific constraints, or share blind spots with the model being evaluated.

The engineering posture should be calibration, not blind trust. Treat the judge as an instrument whose bias, variance, and operating envelope must be measured before its scores are used for deployment decisions.

Calibration protocol

Minimum checks before using automated judging

Check	Method	Risk addressed
Position bias	Swap candidate A/B order and compare outcomes	Judge favors first or second answer
Verbosity bias	Control answer length while preserving correctness	Judge rewards length over quality
Rubric adherence	Use answers that violate one criterion at a time	Judge ignores explicit scoring instructions
Human agreement	Sample expert labels for high-impact tasks	Judge diverges from domain judgment
Self-preference	Judge outputs from same-family and different-family models	Judge favors familiar style
Uncertainty behavior	Require confidence and abstention on ambiguous cases	Judge overstates weak decisions

Judge disagreement matrix

candidate output

rubric

judge A

judge B

human panel sample

disagreement analysis

calibrated scoring policy

Implementation sketch

Store judge decisions as auditable records

judge-record.tsts

type JudgeRecord = {
  sampleId: string;
  rubricVersion: string;
  judgeModel: string;
  candidateOrder: string[];
  scores: Record<string, number>;
  rationale: string;
  confidence: "low" | "medium" | "high";
  requiresHumanReview: boolean;
};

function requiresReview(record: JudgeRecord) {
  return (
    record.confidence === "low" ||
    Math.max(...Object.values(record.scores)) - Math.min(...Object.values(record.scores)) < 0.1
  );
}

For high-impact evaluations, judge rationales should support debugging, but the numeric score should remain traceable to a rubric and calibration sample.

Practical conclusion

Use judges for scale, not final authority

The strongest pattern is hybrid evaluation: objective tests where possible, LLM judges for scalable qualitative triage, and human review for ambiguous, high-impact, or calibration-critical samples. Automated judging should accelerate evidence collection, not remove accountability.

References

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena HELM: Holistic Evaluation of Language Models

Related research

View all

ENT-RN-009