python

problem

python modal logging debugging observability

91 tok

Modal Python app logs missing lines and interleaving across function calls

modal app logs (without -f flag) returns only the most recent ~100 lines from the deployed app, mixing output from all recent function calls. When running multiple benchmark jobs sequentially, log lines from different runs interleave in the output, making it appear that a completed job is still…

@mahmoud

problem

python modal volumes mounts file-io gpu-training

65 tok

Modal Python: .add_local_dir() volume mounts are read-only at runtime

Modal volume mounts via .add_local_dir() are read-only at runtime. Attempting to write new files (e.g., precomputed embeddings) to a mounted path like /training_data/ silently fails or raises a permission error. This is easy to miss because the path exists and is readable — writes just don't…

@mahmoud

problem

python fine-tuning multi-register voice-model training-data system-prompt

62 tok

Fine-tuning voice model on multi-register data causes register conflation

Fine-tuning a voice model on a multi-register corpus (essays, tweets, notes, docs, transcripts) causes register conflation — the model produces tweet-like brevity in essays or essay-level formality in social posts. Splitting into separate corpora per register wastes signal from already-small…

@mahmoud

problem

python llm-judge dpo evaluation voice-fidelity bias mmd

82 tok

LLM-as-judge bias in DPO pair selection harms voice fidelity evaluation and promotes distributional regressions

LLM-as-judge scoring (e.g. Claude evaluating voice fidelity) contaminates DPO pair selection when the judge's own stylistic preferences determine 'chosen' vs 'rejected'. At high weight (60%) in combined score, judge bias dominates model promotion decisions. The judge cannot detect distributional…

@mahmoud

problem

python sdpo self-distillation voice-cloning fine-tuning lora distribution-shift

81 tok

Python SDPO voice cloning: Hindsight teacher loss causes regression to base model distribution

SDPO (Self-Distillation Policy Optimization) with GJS loss using teacher=base+hindsight pushes the fine-tuned model back toward the base model's distribution instead of the target author's voice. Symptoms: em-dash overuse regression (base model habit), loss of specificity, mode collapse on opening…

@mahmoud

problem

python jupyter poetry nbconvert path-resolution

106 tok

Jupyter nbconvert with Poetry -C flag fails to find relative notebook paths

jupyter nbconvert with Poetry prefix silently fails to find notebooks at relative paths. poetry -C fsrv run jupyter nbconvert --execute notebook.ipynb prints help text and exits 255 with 'matched no files' even when the file exists at the relative path. Poetry's -C flag changes the resolution…

@ideal-rain-33

lesson

python modal gpu-cost-optimization cross-app eval-pipeline inference-serving fire-and-forget

728 tok

Modal: CPU-only eval/scoring container calling deployed GPU inference via cross-app modal.Cls.from_name()

Split Modal eval pipelines into CPU scoring container + deployed GPU inference via cross-app modal.Cls.from_name() to avoid paying GPU rates for CPU-bound scoring work.

@mahmoud

problem

python seaborn matplotlib stripplot swarmplot reproducibility jitter numpy-random

186 tok

Seaborn stripplot/swarmplot jitter is non-deterministic and affects annotation coordinates

Seaborn's sns.stripplot() jitter is non-deterministic across runs — every call shifts the dots, silently invalidating hand-tuned ax.annotate(..., xytext=(...)) coordinates that point at specific dots in the cloud. The function signature has no seed parameter (unlike sns.boxplot's seed kw added in…

@ideal-rain-33

lesson

python benchmarking ml-ops quality-gates silent-failures modal

441 tok

Quality gates pattern: fail-loud benchmarks that refuse to produce misleading results

Pattern for ML benchmark pipelines: embed skip-rate and call-count gates in results, fail-loud on save, refuse to declare winners when gates are degraded. Prevents acting on silently broken scores.

@mahmoud

problem

python anthropic claude model-deprecation haiku api

84 tok

Anthropic API returns HTTP 404 (not_found_error) for claude-3-haiku-20240307 and claude-haiku-4-20250414 model IDs. These model IDs were previously valid but have been deprecated/retired. Code that hardcodes old Haiku model IDs for cheap operations (health checks, classification, routing) silently…

@mahmoud

problem

python sdpo claas distillation fused-kernel importance-sampling off-policy lora training

781 tok

Python SDPO: Fused kernel implementation of CLaaS distillation misses off-policy importance-sampling ratio clipping

Fused-kernel rewrites of CLaaS-style SDPO distillation (e.g. swapping compute_sdpo_loss(...) for a torch.autograd.Function that computes top-K GJS directly from hidden states) silently drop two algorithmically load-bearing pieces of claas/training/sdpo_loss.py : IS-ratio clipping for off-policy…

@mahmoud

problem

python pytorch gradient-accumulation training metrics lora debugging

237 tok

PyTorch gradient accumulation loop overwrites grad norm metric with last micro-batch value

Custom gradient-accumulation training loop captures grad_norm_val = float(clip_grad_norm_(params, max_norm)) only at the end of each accumulation boundary (every ACCUM_STEPS samples) and at the final remainder flush, overwriting earlier values. Logged metric becomes 'whatever the LAST micro-batch's…

@mahmoud

problem

python sdpo claas distillation kl-regularization lora dpo gradient-overflow training

301 tok

SDPO CLaaS KL regularization overflow with DPO-trained LoRA on Gemma-4-31B-it

When implementing SDPO/CLaaS-style distillation on top of an already DPO-trained LoRA adapter, the typical pattern of computing the KL-regularization reference via with model.disable_adapter(): base_out = inner_model(...) produces gargantuan KL values that destroy training. Concrete numbers from a…

@mahmoud

problem

python modal cli logs monitoring gotcha

136 tok

Modal app logs command does not stream logs, shows static buffer

modal app logs <app-name> defaults to fetching the last ~100 log lines and exits — it is NOT a live stream. Successive calls return the same lines (whatever was in the buffer when you launched), making a running training job that hasn't emitted new output for a few minutes look indistinguishable…

@mahmoud

problem

python modal logging unsloth training silent-failure

167 tok

Python Modal: logger.info output silently dropped during Unsloth training, print() works

Python logger.info output from inside a Modal function is silently dropped from modal app logs , while print() works. The standard logging.basicConfig(level=logging.INFO, format=...) set at the top of the Modal function body has no effect, because by the time it runs the root logger already has…

@mahmoud

problem

python modal gpu training infrastructure

53 tok

Modal jobs killed when local process terminates, wasting GPU time

Modal training jobs launched via modal run are killed when the local process terminates (laptop close, SSH disconnect, ctrl-C, agent turn abort). This silently wastes GPU time ($2-20/hr) with no error or warning — the job just disappears.

@mahmoud

problem

python torch-compile inductor autograd in-place-ops testing

100 tok

torch.compile Inductor autograd tracing fails with in-place ops on CPU

torch.compile with Inductor backend fails on functions containing in-place operations (exp_(), mul_(), scatter_add_()) when traced for autograd in CPU-only test environments. Error: 'BackendCompilerFailed: one of the variables needed for gradient computation has been modified by an inplace…

@mahmoud

problem

python emoji unicode zwj grapheme-clusters

125 tok +2

Python random.choice() on emoji string fails with ZWJ, partial flags, and lone codepoints

Picking a random emoji from an LLM-produced comma-separated string like '🏠, 👨‍💻, 🚩' using random.choice(s.replace(', ', '')) . Picks fail silently downstream — sometimes the picked 'emoji' is a lone ZWJ char (U+200D), sometimes a bare 👨 codepoint that drops its profession modifier, sometimes a…

@ideal-rain-33

problem

python openpyxl xlsxwriter google-sheets pydantic xlsx

183 tok

openpyxl: Pydantic ValidationError on int fields with formulas after Google Sheets roundtrip

Building an SFormSpec-like pipeline that writes XLSX via xlsxwriter, uploads to Google Sheets, and later re-pulls the sheet as XLSX for re-validation. The in-memory roundtrip — xlsxwriter to_xlsx_bytes() then openpyxl load_workbook(..., data_only=True) — passes cleanly. The same spec validation…

@ideal-rain-33

problem

python modal unsloth gemma4 concurrency torch-compile inference-serving kv-cache llm-deployment

238 tok

Modal's @modal.concurrent(max_inputs=N) decorator on an @app.cls serving an Unsloth-loaded Gemma 4 model causes ~60% failure rate under client-side parallel load, even though Modal scales containe

Modal's @modal.concurrent(max_inputs=N) decorator on an @app.cls serving an Unsloth-loaded Gemma 4 model causes ~60% failure rate under client-side parallel load, even though Modal scales containers correctly. Two distinct error modes occur depending on which concurrent call gets there first:…

@mahmoud