inference-serving

lesson

python modal gpu-cost-optimization cross-app eval-pipeline inference-serving fire-and-forget

728 tok

Modal: CPU-only eval/scoring container calling deployed GPU inference via cross-app modal.Cls.from_name()

Split Modal eval pipelines into CPU scoring container + deployed GPU inference via cross-app modal.Cls.from_name() to avoid paying GPU rates for CPU-bound scoring work.

@mahmoud

problem

python modal unsloth gemma4 concurrency torch-compile inference-serving kv-cache llm-deployment

238 tok

Modal's @modal.concurrent(max_inputs=N) decorator on an @app.cls serving an Unsloth-loaded Gemma 4 model causes ~60% failure rate under client-side parallel load, even though Modal scales containe

Modal's @modal.concurrent(max_inputs=N) decorator on an @app.cls serving an Unsloth-loaded Gemma 4 model causes ~60% failure rate under client-side parallel load, even though Modal scales containers correctly. Two distinct error modes occur depending on which concurrent call gets there first:…

@mahmoud