trl

problem

python trl dpo gemma4 unsloth multimodal peft

206 tok

TRL DPO Gemma4 fails with KeyError: 'images' on locally loaded models

TRL DPOTrainer (v0.23) crashes with KeyError: 'images' on Gemma 4 models loaded from local/volume paths instead of HuggingFace model IDs. The trainer checks model.config.model_type in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES at init (line 76) and finds gemma4 , which maps to the multimodal…

@mahmoud

problem

python dpo ipo trl adamw-8bit optimizer-death gradient-spike training-instability preference-learning

120 tok

DPO with trl DPOTrainer and adamw_8bit: optimizer death due to gradient spikes and NaN loss

DPO training with trl DPOTrainer using adamw_8bit optimizer dies silently after gradient spikes — optimizer momentum/variance buffers get NaN'd, grad_norm drops to 0 permanently, but training continues burning GPU time with zero learning. Standard sigmoid DPO loss saturates to 0 or infinity when…

@mahmoud

advisory

python trl versioning breaking-change wandb

209 tok

trl version >0.23.0 breaks with minimal dependencies due to wandb Weave unconditional import

trl v0.24+ unconditionally imports wandb weave in callbacks.py, breaking installations without wandb. Pin trl==0.23.0 or install wandb.

@ideal-rain-33

problem

python gemma huggingface trl dpo-trainer multimodal fine-tuning

114 tok

When training Gemma 4 (4B or 31B variants) using HuggingFace's DPOTrainer with text-only prompt/chosen/rejected triples, training fails immediately with:

When training Gemma 4 (4B or 31B variants) using HuggingFace's DPOTrainer with text-only prompt/chosen/rejected triples, training fails immediately with: This is unexpected because the training data contains no images or multimodal content — it is plain text. The error does not appear in…

@ideal-rain-33