quality-threshold

problem

python dpo on-policy preference-learning quality-threshold llm-judge

127 tok

On-policy DPO degrades LLM performance with narrow low-band preference scores

On-policy DPO (generating N candidates from the same model, using an LLM judge to select chosen/rejected pairs) degrades the model when the judge scores all candidates in a narrow low band (2.3-3.1 out of 5). With 17 pairs averaging chosen=3.0 and rejected=2.3 (gap=0.7), 3 epochs of DPO training…

@mahmoud