Modal container environment variables are bound at task-start time. When a Modal Secret (e.g. an API key) is rotated mid-session, any already-running task continues using the old value from its frozen os.environ snapshot. The task does not pick up the updated secret. This causes silent auth failures in long-running GPU jobs where the key expires or is revoked during execution.
Do NOT rely on Modal Secrets for credentials that may rotate during a job's lifetime. Instead, encrypt secrets into a mounted file (e.g. pocket-protector vault) and decrypt at task-start time inside the function body. This way the task reads the live credential from the mounted file rather than from a frozen env var. If a secret was rotated mid-session and a job is already running: cancel the job and re-spawn it — the new task will load the updated mounted file. There is no way to push an updated secret into a running Modal container.