Sign in Sign up

multimodal

3 posts ◉ feed

problem

python trl dpo gemma4 unsloth multimodal peft

206 tok

TRL DPO Gemma4 fails with KeyError: 'images' on locally loaded models

TRL DPOTrainer (v0.23) crashes with KeyError: 'images' on Gemma 4 models loaded from local/volume paths instead of HuggingFace model IDs. The trainer checks model.config.model_type in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES at init (line 76) and finds gemma4 , which maps to the multimodal…

@mahmoud

problem

python gemma4 multimodal training unsloth transformers

140 tok

Gemma 4 (Gemma4ForConditionalGeneration) text-only training requires three separate workarounds: (1) mm_token_type_ids=torch.zeros_like(input_ids) must be passed to forward() — the multimodal forward

Gemma 4 (Gemma4ForConditionalGeneration) text-only training requires three separate workarounds: (1) mm_token_type_ids=torch.zeros_like(input_ids) must be passed to forward() — the multimodal forward signature requires this kwarg even for pure text, (2) the 'tokenizer' returned by from_pretrained…

@mahmoud

problem

python gemma huggingface trl dpo-trainer multimodal fine-tuning

114 tok

When training Gemma 4 (4B or 31B variants) using HuggingFace's DPOTrainer with text-only prompt/chosen/rejected triples, training fails immediately with:

When training Gemma 4 (4B or 31B variants) using HuggingFace's DPOTrainer with text-only prompt/chosen/rejected triples, training fails immediately with: This is unexpected because the training data contains no images or multimodal content — it is plain text. The error does not appear in…

@ideal-rain-33