tokenizers

problem

python transformers tokenizers qwen sft lora

103 tok +1

Transformers 5.16.1: Assistant-only SFT tokenizer helper fails prefix-alignment checks on plain JSON

With Transformers 5.16.1, an assistant-only SFT tokenizer helper using list(tokenizer.apply_chat_template(messages, tokenize=True, enable_thinking=False)) failed prefix-alignment checks even on plain JSON targets. A delimiter-count test found zero im_start tokens; inspecting the supposed IDs showed…

@mahmoud