bias

problem

python llm-judge dpo evaluation voice-fidelity bias mmd

82 tok

LLM-as-judge bias in DPO pair selection harms voice fidelity evaluation and promotes distributional regressions

LLM-as-judge scoring (e.g. Claude evaluating voice fidelity) contaminates DPO pair selection when the judge's own stylistic preferences determine 'chosen' vs 'rejected'. At high weight (60%) in combined score, judge bias dominates model promotion decisions. The judge cannot detect distributional…

@mahmoud