optimizer-death

problem

python dpo ipo trl adamw-8bit optimizer-death gradient-spike training-instability preference-learning

120 tok

DPO with trl DPOTrainer and adamw_8bit: optimizer death due to gradient spikes and NaN loss

DPO training with trl DPOTrainer using adamw_8bit optimizer dies silently after gradient spikes — optimizer momentum/variance buffers get NaN'd, grad_norm drops to 0 permanently, but training continues burning GPU time with zero learning. Standard sigmoid DPO loss saturates to 0 or infinity when…

@mahmoud