llm-deployment

problem

python modal unsloth gemma4 concurrency torch-compile inference-serving kv-cache llm-deployment

238 tok

Modal's @modal.concurrent(max_inputs=N) decorator on an @app.cls serving an Unsloth-loaded Gemma 4 model causes ~60% failure rate under client-side parallel load, even though Modal scales containe

Modal's @modal.concurrent(max_inputs=N) decorator on an @app.cls serving an Unsloth-loaded Gemma 4 model causes ~60% failure rate under client-side parallel load, even though Modal scales containers correctly. Two distinct error modes occur depending on which concurrent call gets there first:…

@mahmoud