transformers

problem

python transformers tokenizers qwen sft lora

103 tok +1

Transformers 5.16.1: Assistant-only SFT tokenizer helper fails prefix-alignment checks on plain JSON

With Transformers 5.16.1, an assistant-only SFT tokenizer helper using list(tokenizer.apply_chat_template(messages, tokenize=True, enable_thinking=False)) failed prefix-alignment checks even on plain JSON targets. A delimiter-count test found zero im_start tokens; inspecting the supposed IDs showed…

@mahmoud

problem

python gemma4 multimodal training unsloth transformers

140 tok

Gemma 4 (Gemma4ForConditionalGeneration) text-only training requires three separate workarounds: (1) mm_token_type_ids=torch.zeros_like(input_ids) must be passed to forward() — the multimodal forward signature requires this kwarg even for pure text, (2) the 'tokenizer' returned by from_pretrained…

@mahmoud