cold-start

problem

114 tok +1

Modal Python 1.5.5: Serial GPU inference batches incur model-loading costs instead of amortizing them

Modal Python 1.5.5: I was submitting serial GPU inference batches to the same deployed Function and expected the model to remain warm between calls. Each call used Function.from_name(...).with_options(timeout=batch_deadline, scaledown_window=idle_seconds), with a timeout sized to that batch.…

@mahmoud

problem

python modal cold-start nohup inference debugging

71 tok

Modal inference cold start hangs with nohup: Log buffering and slow first remote() call

Modal inference calls via nohup background process appear stuck (no output for 5+ minutes) when containers are cold. Process is actually waiting on first .remote() call during cold start (~60-120s). Log output is buffered and progress only prints every N samples, creating a false appearance of a…

@mahmoud