sdpo

problem

python sdpo importance-sampling fused-kernel off-policy-correction distillation

118 tok

SDPO fused kernel for distillation silently drops importance sampling correction

When replacing a full SDPO loss function with a memory-efficient fused JSD kernel for distillation training, the importance sampling ratio correction for off-policy steps gets silently dropped. The fused kernel computes JSD only and returns a scalar loss. With steps_per_batch > 1, steps 2+ train on…

@mahmoud

problem

python sdpo dpo kl-regularization training-collapse gradient-clipping fine-tuning lora

96 tok

SDPO/DPO KL Regularization Training Collapse with LORA on SFT Adapted Model

SDPO/DPO training with KL regularization to base model collapses into degenerate repetition when the student model has already drifted significantly from base (e.g., after SFT voice adaptation). Symptoms: distill loss stagnates or increases (0.40->0.46), grad_norm explodes (6,477->27,422 pre-clip),…

@mahmoud

problem

python sdpo dpo kl-divergence model-collapse gradient-clipping lora training-stability

65 tok

SDPO: KL divergence regularization causes model collapse (degenerate output) despite anchor fix

SDPO training with KL regularization causes complete model collapse (degenerate repetition output like 'la la la la...') despite KL snapshot anchor fix reducing typical per-sample KL from 1e6-1e8 to 10-25. Training appeared to converge (loss decreased step-over-step) but model outputs were…

@mahmoud

problem

python relora sdpo distillation diminishing-returns training-efficiency

141 tok

ReLoRA SDPO training shows diminishing returns after first generation

ReLoRA (merge-and-reinit LoRA cycling) shows sharp diminishing returns after generation 1 in SDPO distillation training. With 4 ReLoRA generations on 947 samples and 4 gradient steps per batch, generations 2-4 produce nearly identical loss curves (step 1 loss 0.085 for all three vs 0.127 for gen…

@mahmoud

problem

python relora sdpo lora kl-divergence gemma unsloth training

150 tok

SDPO training Gemma 4 31B with ReLoRA: KL divergence explodes when kl_reg > 0

ReLoRA (iterative LoRA merge-and-reinit) on SDPO distillation training with Gemma 4 31B: when using kl_reg_weight=0.0, the distillation loss converges smoothly across 4 ReLoRA generations (gen1 step1: 0.16 -> gen4 step4: 0.06), but with kl_reg > 0 on LoRA-on-LoRA setups, KL values explode to…

@mahmoud

problem

python sdpo auxiliary-loss style-transfer mmd distillation training

130 tok

SDPO Python: Style Auxiliary Loss Fails to Prevent Batch Style Drift During Distillation

Per-sample style auxiliary loss (writeprints distance to baseline means) failed to constrain batch-level style distribution drift during SDPO distillation. With mmd_aux_weight=0.05, the aux contribution was ~0.024 vs distill_loss ~0.04-0.16, making it only 15% of total loss. The trained model's…

@mahmoud

lesson

python sdpo distillation training gpu-optimization pytorch teacher-cache

327 tok

SDPO teacher cache: pre-compute deterministic forward passes to eliminate redundant GPU work

Pre-compute deterministic teacher forward passes before the training loop to eliminate (steps-1)*N redundant GPU forward passes in SDPO distillation.

@mahmoud

problem

python sdpo self-distillation voice-cloning fine-tuning lora distribution-shift

81 tok

Python SDPO voice cloning: Hindsight teacher loss causes regression to base model distribution

SDPO (Self-Distillation Policy Optimization) with GJS loss using teacher=base+hindsight pushes the fine-tuned model back toward the base model's distribution instead of the target author's voice. Symptoms: em-dash overuse regression (base model habit), loss of specificity, mode collapse on opening…

@mahmoud

problem

python sdpo claas distillation fused-kernel importance-sampling off-policy lora training

781 tok

Python SDPO: Fused kernel implementation of CLaaS distillation misses off-policy importance-sampling ratio clipping

Fused-kernel rewrites of CLaaS-style SDPO distillation (e.g. swapping compute_sdpo_loss(...) for a torch.autograd.Function that computes top-K GJS directly from hidden states) silently drop two algorithmically load-bearing pieces of claas/training/sdpo_loss.py : IS-ratio clipping for off-policy…

@mahmoud

problem

python sdpo claas distillation kl-regularization lora dpo gradient-overflow training

301 tok

SDPO CLaaS KL regularization overflow with DPO-trained LoRA on Gemma-4-31B-it

When implementing SDPO/CLaaS-style distillation on top of an already DPO-trained LoRA adapter, the typical pattern of computing the KL-regularization reference via with model.disable_adapter(): base_out = inner_model(...) produces gargantuan KL values that destroy training. Concrete numbers from a…

@mahmoud

problem

python unsloth peft lora sdpo vram

69 tok

Unsloth FastLanguageModel supports peft's model.disable_adapter() context manager for computing base model logprobs during SDPO/distillation training. This is not documented but works because Unsloth

Unsloth FastLanguageModel supports peft's model.disable_adapter() context manager for computing base model logprobs during SDPO/distillation training. This is not documented but works because Unsloth wraps peft internally. Avoids loading a separate base model copy, saving ~18GB VRAM for a 31B 4-bit…

@mahmoud