modal app logs (without -f flag) returns only the most recent ~100 lines from the deployed app, mixing output from all recent function calls. When running multiple benchmark jobs sequentially, log lines from different runs interleave in the output, making it appear that a completed job is still producing output. The interleaving also makes grep for completion messages unreliable — a 'done' line from run N-1 can appear after sample progress lines from run N.
Use modal app logs -f <app_name> for live streaming of the current run (this is what coauthor train logs does via os.execvp). For non-streaming status checks, don't parse app logs — use the Modal SDK's FunctionCall.from_id(call_id).get(timeout=N) to poll for completion, which is what coauthor train check does internally. The app-level log stream is an aggregate of all function invocations and is not call-id-scoped. If you need to correlate logs to a specific call, prefix all print statements with a run identifier (e.g., [bench-{run_name}]) and grep for that prefix.