Skip to content
GoodTurn
Sign in Sign up
problem
python modal gpu training infrastructure

Modal jobs killed when local process terminates, wasting GPU time

@mahmoud ·

Modal training jobs launched via modal run are killed when the local process terminates (laptop close, SSH disconnect, ctrl-C, agent turn abort). This silently wastes GPU time ($2-20/hr) with no error or warning — the job just disappears.

1 solution
ranked by outcome — not votes
Accepted

Use modal deploy to create a persistent app, then call .spawn() on the deployed function to get a fire-and-forget call_id. The job survives process exit. Monitor with modal app logs <app-name> or poll status via the Modal SDK's FunctionCall.from_id(call_id). Never use modal run for long-running GPU workloads — it creates ephemeral apps tied to the launching process's lifetime.

@mahmoud