Modal Python 1.5.5: Serial GPU inference batches incur model-loading costs instead of amortizing them
Modal Python 1.5.5: I was submitting serial GPU inference batches to the same deployed Function and expected the model to remain warm between calls. Each call used Function.from_name(...).with_options(timeout=batch_deadline, scaledown_window=idle_seconds), with a timeout sized to that batch. Breaking work into smaller bounded batches repeatedly incurred model-loading costs instead of amortizing them, and several first cases exhausted their loading/generation allowance. The deployed function name and model were unchanged; there was no warning that these invocations could not share their previous warm containers.
Function.with_options() creates a dynamically configured Function whose invocations run in a distinct container pool, separate both from the base function and from other dynamic configurations. The timeout is part of that configuration. Extending scaledown_window cannot make different timeout configurations share a pool.
import modal
base = modal.Function.from_name("inference-app", "generate")
a = base.with_options(timeout=600, scaledown_window=120)
b = base.with_options(timeout=900, scaledown_window=120)
# a and b use separate pools, although only timeout differs.For intended reuse, keep the dynamic configuration identical and pass any narrower application deadline as a payload value. Reuse is still an optimization, never guaranteed by the autoscaler. If different provider timeouts are required, budget a cold start for each configuration. In the observed evaluation, the chosen approach was one measured, bounded session per checkpoint instead of many small sessions. Loading had its own allowance rather than consuming the entire decoding deadline, while the whole session retained its hard provider timeout and unchanged output caps.
Evidence: serial evaluation batches with varying runtime options repeatedly loaded the same model; the official Function API explicitly states that each dynamic configuration runs in a separate, independently autoscaled pool: https://modal.com/docs/sdk/py/latest/Function#with_options. This is pool identity behavior, not a missing model-cache entry or evidence that scaledown_window was ignored.