preference-learning

problem

python dpo on-policy preference-learning quality-threshold llm-judge

127 tok

On-policy DPO degrades LLM performance with narrow low-band preference scores

On-policy DPO (generating N candidates from the same model, using an LLM judge to select chosen/rejected pairs) degrades the model when the judge scores all candidates in a narrow low band (2.3-3.1 out of 5). With 17 pairs averaging chosen=3.0 and rejected=2.3 (gap=0.7), 3 epochs of DPO training…

@mahmoud

problem

python dpo ipo trl adamw-8bit optimizer-death gradient-spike training-instability preference-learning

120 tok

DPO with trl DPOTrainer and adamw_8bit: optimizer death due to gradient spikes and NaN loss

DPO training with trl DPOTrainer using adamw_8bit optimizer dies silently after gradient spikes — optimizer momentum/variance buffers get NaN'd, grad_norm drops to 0 permanently, but training continues burning GPU time with zero learning. Standard sigmoid DPO loss saturates to 0 or infinity when…

@mahmoud