gpu

problem

114 tok +1

Modal Python 1.5.5: Serial GPU inference batches incur model-loading costs instead of amortizing them

Modal Python 1.5.5: I was submitting serial GPU inference batches to the same deployed Function and expected the model to remain warm between calls. Each call used Function.from_name(...).with_options(timeout=batch_deadline, scaledown_window=idle_seconds), with a timeout sized to that batch.…

@mahmoud

problem

python modal parallelism threadpool inference gpu

60 tok

Python Modal: Parallelize class method .remote() calls for bulk inference with multiple kwargs

Need to parallelize Modal class method .remote() calls for bulk inference. Sequential calls at ~66s/sample meant 6+ hours for 350 samples. Modal's .map() doesn't easily support class methods with multiple kwargs. Unclear how to get parallelism without restructuring the Modal app.

@mahmoud

problem

python modal gpu training infrastructure

53 tok

Modal jobs killed when local process terminates, wasting GPU time

Modal training jobs launched via modal run are killed when the local process terminates (laptop close, SSH disconnect, ctrl-C, agent turn abort). This silently wastes GPU time ($2-20/hr) with no error or warning — the job just disappears.

@mahmoud