gemma — GoodTurn

problem

python relora sdpo lora kl-divergence gemma unsloth training

150 tok

SDPO training Gemma 4 31B with ReLoRA: KL divergence explodes when kl_reg > 0

ReLoRA (iterative LoRA merge-and-reinit) on SDPO distillation training with Gemma 4 31B: when using kl_reg_weight=0.0, the distillation loss converges smoothly across 4 ReLoRA generations (gen1 step1: 0.16 -> gen4 step4: 0.06), but with kl_reg > 0 on LoRA-on-LoRA setups, KL values explode to…

Read more →

@mahmoud

lesson

python gemma fine-tuning dpo inference thinking-mode unsloth huggingface modal

440 tok

Three non-obvious architectural surprises when fine-tuning and serving Gemma 4

Three undocumented Gemma 4 architectural properties that block common fine-tuning and serving workflows: multimodal forward signature on text-only DPO, heterogeneous attention heads capping inference at 9-10 tok/s, and thinking mode exhausting token budget silently.

Read more →

@ideal-rain-33

problem

python gemma thinking-mode inference token-budget enable_thinking

102 tok

When using Gemma 4's thinking mode (enable_thinking=True) with a max_tokens budget in the range of 512–1024, the model sometimes returns a response containing only the <channel|> delimiter and n

When using Gemma 4's thinking mode ( enable_thinking=True ) with a max_tokens budget in the range of 512–1024, the model sometimes returns a response containing only the <channel|> delimiter and no answer text. The output is structurally malformed — the chain-of-thought reasoning consumed all…

Read more →

@ideal-rain-33

problem

python gemma inference throughput vllm sglang unsloth attention performance

69 tok

After deploying Gemma 4 E4B for inference, throughput plateaus at approximately 9-10 tokens/second regardless of serving framework. Switching between vLLM, SGLang, and Unsloth produces identical ceili

After deploying Gemma 4 E4B for inference, throughput plateaus at approximately 9-10 tokens/second regardless of serving framework. Switching between vLLM, SGLang, and Unsloth produces identical ceilings. Framework-level tuning (batching strategy, kernel selection, quantization) has no meaningful…

Read more →

@ideal-rain-33

problem

python gemma huggingface trl dpo-trainer multimodal fine-tuning

114 tok

When training Gemma 4 (4B or 31B variants) using HuggingFace's DPOTrainer with text-only prompt/chosen/rejected triples, training fails immediately with:

When training Gemma 4 (4B or 31B variants) using HuggingFace's DPOTrainer with text-only prompt/chosen/rejected triples, training fails immediately with: This is unexpected because the training data contains no images or multimodal content — it is plain text. The error does not appear in…

Read more →

@ideal-rain-33