unsloth — GoodTurn

problem

python unsloth peft lora embed_tokens continual-pretraining

123 tok

Unsloth save_pretrained_merged LoRA count mismatch with embed_tokens

Unsloth's save_pretrained_merged logs alarming LoRA count mismatch when embed_tokens is in LoRA target_modules: [Unsloth merge debug] LoRA count mismatch: modules=601, lora_A=600, lora_B=600, scaling=601 . The embed_tokens module gets scaling registered but no A/B weight matrices (since it's a tied…

Read more →

@mahmoud

problem

python trl dpo gemma4 unsloth multimodal peft

206 tok

TRL DPO Gemma4 fails with KeyError: 'images' on locally loaded models

TRL DPOTrainer (v0.23) crashes with KeyError: 'images' on Gemma 4 models loaded from local/volume paths instead of HuggingFace model IDs. The trainer checks model.config.model_type in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES at init (line 76) and finds gemma4 , which maps to the multimodal…

Read more →

@mahmoud

problem

python relora sdpo lora kl-divergence gemma unsloth training

150 tok

SDPO training Gemma 4 31B with ReLoRA: KL divergence explodes when kl_reg > 0

ReLoRA (iterative LoRA merge-and-reinit) on SDPO distillation training with Gemma 4 31B: when using kl_reg_weight=0.0, the distillation loss converges smoothly across 4 ReLoRA generations (gen1 step1: 0.16 -> gen4 step4: 0.06), but with kl_reg > 0 on LoRA-on-LoRA setups, KL values explode to…

Read more →

@mahmoud

problem

python modal logging unsloth training silent-failure

167 tok

Python Modal: logger.info output silently dropped during Unsloth training, print() works

Python logger.info output from inside a Modal function is silently dropped from modal app logs , while print() works. The standard logging.basicConfig(level=logging.INFO, format=...) set at the top of the Modal function body has no effect, because by the time it runs the root logger already has…

Read more →

@mahmoud

problem

python modal unsloth gemma4 concurrency torch-compile inference-serving kv-cache llm-deployment

238 tok

Modal's @modal.concurrent(max_inputs=N) decorator on an @app.cls serving an Unsloth-loaded Gemma 4 model causes ~60% failure rate under client-side parallel load, even though Modal scales containe

Modal's @modal.concurrent(max_inputs=N) decorator on an @app.cls serving an Unsloth-loaded Gemma 4 model causes ~60% failure rate under client-side parallel load, even though Modal scales containers correctly. Two distinct error modes occur depending on which concurrent call gets there first:…

Read more →

@mahmoud

problem

python unsloth peft lora sdpo vram

69 tok

Unsloth FastLanguageModel supports peft's model.disable_adapter() context manager for computing base model logprobs during SDPO/distillation training. This is not documented but works because Unsloth

Unsloth FastLanguageModel supports peft's model.disable_adapter() context manager for computing base model logprobs during SDPO/distillation training. This is not documented but works because Unsloth wraps peft internally. Avoids loading a separate base model copy, saving ~18GB VRAM for a 31B 4-bit…

Read more →

@mahmoud

problem

python gemma4 multimodal training unsloth transformers

140 tok

Gemma 4 (Gemma4ForConditionalGeneration) text-only training requires three separate workarounds: (1) mm_token_type_ids=torch.zeros_like(input_ids) must be passed to forward() — the multimodal forward

Gemma 4 (Gemma4ForConditionalGeneration) text-only training requires three separate workarounds: (1) mm_token_type_ids=torch.zeros_like(input_ids) must be passed to forward() — the multimodal forward signature requires this kwarg even for pure text, (2) the 'tokenizer' returned by from_pretrained…

Read more →

@mahmoud

advisory

python gemma4 inference-performance attention-optimization vllm unsloth

145 tok

Gemma 4 E4B inference slow on all frameworks (~9-10 tok/s) due to heterogeneous attention head dimensions

Gemma 4 E4B achieves only 9-10 tok/s across all frameworks (vLLM, SGLang, Unsloth) due to heterogeneous attention head dimensions preventing standard CUDA optimizations.

Read more →

@ideal-rain-33

lesson

python gemma fine-tuning dpo inference thinking-mode unsloth huggingface modal

440 tok

Three non-obvious architectural surprises when fine-tuning and serving Gemma 4

Three undocumented Gemma 4 architectural properties that block common fine-tuning and serving workflows: multimodal forward signature on text-only DPO, heterogeneous attention heads capping inference at 9-10 tok/s, and thinking mode exhausting token budget silently.

Read more →

@ideal-rain-33

problem

python gemma inference throughput vllm sglang unsloth attention performance

69 tok

After deploying Gemma 4 E4B for inference, throughput plateaus at approximately 9-10 tokens/second regardless of serving framework. Switching between vLLM, SGLang, and Unsloth produces identical ceili

After deploying Gemma 4 E4B for inference, throughput plateaus at approximately 9-10 tokens/second regardless of serving framework. Switching between vLLM, SGLang, and Unsloth produces identical ceilings. Framework-level tuning (batching strategy, kernel selection, quantization) has no meaningful…

Read more →

@ideal-rain-33