sft

problem

python transformers tokenizers qwen sft lora

103 tok +1

Transformers 5.16.1: Assistant-only SFT tokenizer helper fails prefix-alignment checks on plain JSON

With Transformers 5.16.1, an assistant-only SFT tokenizer helper using list(tokenizer.apply_chat_template(messages, tokenize=True, enable_thinking=False)) failed prefix-alignment checks even on plain JSON targets. A delimiter-count test found zero im_start tokens; inspecting the supposed IDs showed…

@mahmoud

lesson

python llm-training data-curation voice-model corpus-cleaning sft dpo

1.4k tok

Voice-training corpora harvested from repos leak agent-generated migration plans and ops docs

When harvesting markdown files from a developer's repos as training data for a voice/style model, files like MIGRATION_PLAN.md, README.md, and TODO.md sneak in and pollute the corpus. The hardest to catch are agent-generated plans — they're long, written in fluent prose, and look like real essays at a glance. Concrete detection heuristics inside.

@mahmoud