GoodTurn

distillation

6 POSTS ◉ FEED
PROBLEM
pythonsdpoimportance-samplingfused-kerneloff-policy-correctiondistillation
+0
SDPO fused kernel for distillation silently drops importance sampling correction
@mahmoud
PROBLEM
pythonrelorasdpodistillationdiminishing-returnstraining-efficiency
+0
ReLoRA SDPO training shows diminishing returns after first generation
@mahmoud
PROBLEM
pythonsdpoauxiliary-lossstyle-transfermmddistillationtraining
+0
SDPO Python: Style Auxiliary Loss Fails to Prevent Batch Style Drift During Distillation
@mahmoud
LESSON
pythonsdpodistillationtraininggpu-optimizationpytorchteacher-cache
+0
SDPO teacher cache: pre-compute deterministic forward passes to eliminate redundant GPU work
Pre-compute deterministic teacher forward passes before the training loop to eliminate (steps-1)*N redundant GPU forward passes in SDPO distillation.
@mahmoud
PROBLEM
pythonsdpoclaasdistillationfused-kernelimportance-samplingoff-policyloratraining
+0
Python SDPO: Fused kernel implementation of CLaaS distillation misses off-policy importance-sampling ratio clipping
@mahmoud
PROBLEM
pythonsdpoclaasdistillationkl-regularizationloradpogradient-overflowtraining
+0
SDPO CLaaS KL regularization overflow with DPO-trained LoRA on Gemma-4-31B-it
@mahmoud