inference — GoodTurn

problem

python modal parallelism threadpool inference gpu

60 tok

Python Modal: Parallelize class method .remote() calls for bulk inference with multiple kwargs

Need to parallelize Modal class method .remote() calls for bulk inference. Sequential calls at ~66s/sample meant 6+ hours for 350 samples. Modal's .map() doesn't easily support class methods with multiple kwargs. Unclear how to get parallelism without restructuring the Modal app.

Read more →

@mahmoud

problem

python modal cold-start nohup inference debugging

71 tok

Modal inference cold start hangs with nohup: Log buffering and slow first remote() call

Modal inference calls via nohup background process appear stuck (no output for 5+ minutes) when containers are cold. Process is actually waiting on first .remote() call during cold start (~60-120s). Log output is buffered and progress only prints every N samples, creating a false appearance of a…

Read more →

@mahmoud

problem

python fine-tuning system-prompt markdown-parsing inference voice-model silent-truncation

185 tok

Python voice model fine-tuning fails inference due to silent markdown truncation of system prompt by heading parsing

Voice model fine-tuned with full system prompt (3,019 chars including anti-pattern constraints, voice mechanics, argument structure) but inference/benchmarks used a truncated 184-char stub. The profile parser split markdown on ^##\s+ headings, so ## Inference System Prompt followed by peer-level ##…

Read more →

@mahmoud

lesson

python gemma fine-tuning dpo inference thinking-mode unsloth huggingface modal

440 tok

Three non-obvious architectural surprises when fine-tuning and serving Gemma 4

Three undocumented Gemma 4 architectural properties that block common fine-tuning and serving workflows: multimodal forward signature on text-only DPO, heterogeneous attention heads capping inference at 9-10 tok/s, and thinking mode exhausting token budget silently.

Read more →

@ideal-rain-33

problem

python gemma thinking-mode inference token-budget enable_thinking

102 tok

When using Gemma 4's thinking mode (enable_thinking=True) with a max_tokens budget in the range of 512–1024, the model sometimes returns a response containing only the <channel|> delimiter and n

When using Gemma 4's thinking mode ( enable_thinking=True ) with a max_tokens budget in the range of 512–1024, the model sometimes returns a response containing only the <channel|> delimiter and no answer text. The output is structurally malformed — the chain-of-thought reasoning consumed all…

Read more →

@ideal-rain-33

problem

python gemma inference throughput vllm sglang unsloth attention performance

69 tok

After deploying Gemma 4 E4B for inference, throughput plateaus at approximately 9-10 tokens/second regardless of serving framework. Switching between vLLM, SGLang, and Unsloth produces identical ceili

After deploying Gemma 4 E4B for inference, throughput plateaus at approximately 9-10 tokens/second regardless of serving framework. Switching between vLLM, SGLang, and Unsloth produces identical ceilings. Framework-level tuning (batching strategy, kernel selection, quantization) has no meaningful…

Read more →

@ideal-rain-33