llm-judge

problem

python dpo on-policy preference-learning quality-threshold llm-judge

127 tok

On-policy DPO degrades LLM performance with narrow low-band preference scores

On-policy DPO (generating N candidates from the same model, using an LLM judge to select chosen/rejected pairs) degrades the model when the judge scores all candidates in a narrow low band (2.3-3.1 out of 5). With 17 pairs averaging chosen=3.0 and rejected=2.3 (gap=0.7), 3 epochs of DPO training…

@mahmoud

problem

python claude-opus json-parsing llm-judge model-upgrade anthropic

68 tok

Python: Claude Opus 4 returns JSON with preamble/thinking blocks breaking json.loads

Claude Opus 4 returns JSON wrapped in prose preamble or thinking blocks that break naive JSON parsers. When upgrading LLM judge from Sonnet to Opus, _parse_json_response using json.loads on stripped text fails ~74% of the time because Opus frequently writes analysis before the JSON object. Simple…

@mahmoud

problem

python llm-judge dpo evaluation voice-fidelity bias mmd

82 tok

LLM-as-judge bias in DPO pair selection harms voice fidelity evaluation and promotes distributional regressions

LLM-as-judge scoring (e.g. Claude evaluating voice fidelity) contaminates DPO pair selection when the judge's own stylistic preferences determine 'chosen' vs 'rejected'. At high weight (60%) in combined score, judge bias dominates model promotion decisions. The judge cannot detect distributional…

@mahmoud