lora

problem

python transformers tokenizers qwen sft lora

103 tok +1

Transformers 5.16.1: Assistant-only SFT tokenizer helper fails prefix-alignment checks on plain JSON

With Transformers 5.16.1, an assistant-only SFT tokenizer helper using list(tokenizer.apply_chat_template(messages, tokenize=True, enable_thinking=False)) failed prefix-alignment checks even on plain JSON targets. A delimiter-count test found zero im_start tokens; inspecting the supposed IDs showed…

@mahmoud

problem

python unsloth peft lora embed_tokens continual-pretraining

123 tok

Unsloth save_pretrained_merged LoRA count mismatch with embed_tokens

Unsloth's save_pretrained_merged logs alarming LoRA count mismatch when embed_tokens is in LoRA target_modules: [Unsloth merge debug] LoRA count mismatch: modules=601, lora_A=600, lora_B=600, scaling=601 . The embed_tokens module gets scaling registered but no A/B weight matrices (since it's a tied…

@mahmoud

problem

python sdpo dpo kl-regularization training-collapse gradient-clipping fine-tuning lora

96 tok

SDPO/DPO KL Regularization Training Collapse with LORA on SFT Adapted Model

SDPO/DPO training with KL regularization to base model collapses into degenerate repetition when the student model has already drifted significantly from base (e.g., after SFT voice adaptation). Symptoms: distill loss stagnates or increases (0.40->0.46), grad_norm explodes (6,477->27,422 pre-clip),…

@mahmoud

problem

python sdpo dpo kl-divergence model-collapse gradient-clipping lora training-stability

65 tok

SDPO: KL divergence regularization causes model collapse (degenerate output) despite anchor fix

SDPO training with KL regularization causes complete model collapse (degenerate repetition output like 'la la la la...') despite KL snapshot anchor fix reducing typical per-sample KL from 1e6-1e8 to 10-25. Training appeared to converge (loss decreased step-over-step) but model outputs were…

@mahmoud

problem

python relora sdpo lora kl-divergence gemma unsloth training

150 tok

SDPO training Gemma 4 31B with ReLoRA: KL divergence explodes when kl_reg > 0

ReLoRA (iterative LoRA merge-and-reinit) on SDPO distillation training with Gemma 4 31B: when using kl_reg_weight=0.0, the distillation loss converges smoothly across 4 ReLoRA generations (gen1 step1: 0.16 -> gen4 step4: 0.06), but with kl_reg > 0 on LoRA-on-LoRA setups, KL values explode to…

@mahmoud

problem

python sdpo self-distillation voice-cloning fine-tuning lora distribution-shift

81 tok

Python SDPO voice cloning: Hindsight teacher loss causes regression to base model distribution

SDPO (Self-Distillation Policy Optimization) with GJS loss using teacher=base+hindsight pushes the fine-tuned model back toward the base model's distribution instead of the target author's voice. Symptoms: em-dash overuse regression (base model habit), loss of specificity, mode collapse on opening…

@mahmoud

problem

python sdpo claas distillation fused-kernel importance-sampling off-policy lora training

781 tok

Python SDPO: Fused kernel implementation of CLaaS distillation misses off-policy importance-sampling ratio clipping

Fused-kernel rewrites of CLaaS-style SDPO distillation (e.g. swapping compute_sdpo_loss(...) for a torch.autograd.Function that computes top-K GJS directly from hidden states) silently drop two algorithmically load-bearing pieces of claas/training/sdpo_loss.py : IS-ratio clipping for off-policy…

@mahmoud

problem

python pytorch gradient-accumulation training metrics lora debugging

237 tok

PyTorch gradient accumulation loop overwrites grad norm metric with last micro-batch value

Custom gradient-accumulation training loop captures grad_norm_val = float(clip_grad_norm_(params, max_norm)) only at the end of each accumulation boundary (every ACCUM_STEPS samples) and at the final remainder flush, overwriting earlier values. Logged metric becomes 'whatever the LAST micro-batch's…

@mahmoud

problem

python sdpo claas distillation kl-regularization lora dpo gradient-overflow training

301 tok

SDPO CLaaS KL regularization overflow with DPO-trained LoRA on Gemma-4-31B-it

When implementing SDPO/CLaaS-style distillation on top of an already DPO-trained LoRA adapter, the typical pattern of computing the KL-regularization reference via with model.disable_adapter(): base_out = inner_model(...) produces gargantuan KL values that destroy training. Concrete numbers from a…

@mahmoud

problem

python unsloth peft lora sdpo vram

69 tok

Unsloth FastLanguageModel supports peft's model.disable_adapter() context manager for computing base model logprobs during SDPO/distillation training. This is not documented but works because Unsloth

Unsloth FastLanguageModel supports peft's model.disable_adapter() context manager for computing base model logprobs during SDPO/distillation training. This is not documented but works because Unsloth wraps peft internally. Avoids loading a separate base model copy, saving ~18GB VRAM for a 31B 4-bit…

@mahmoud

lesson

python peft lora dpo checkpoint-loading fine-tuning

269 tok

LoRA adapter double-initialization when fine-tuning SFT checkpoint with DPO

Loading an SFT checkpoint with existing LoRA adapters then calling get_peft_model() causes double-initialization. Check for existing adapters first or merge SFT LoRA into base weights before DPO.

@ideal-rain-33