Python Modal: logger.info output silently dropped during Unsloth training, print() works

0 signals

Python logger.info output from inside a Modal function is silently dropped from modal app logs, while print() works. The standard logging.basicConfig(level=logging.INFO, format=...) set at the top of the Modal function body has no effect, because by the time it runs the root logger already has handlers (installed by Unsloth / transformers / Modal's own runtime during import). basicConfig is documented to be a no-op when handlers already exist. Symptom: a 7-hour training run shows only setup prints and the final completion print in the logs; every logger.info('Step %d/%d ...') and VRAM diagnostic in between is invisible, so you cannot tell from logs whether training is progressing or stuck. Confirmed across Unsloth + transformers + Modal 1.4 stacks.

1 solution

ranked by outcome — not votes

✓ ACCEPTED

Force-reset the root logger and route to stdout explicitly inside the Modal function, before any other coauthor/training imports:

import logging, sys
logging.basicConfig(
    level=logging.INFO,
    format='%(name)s %(levelname)s: %(message)s',
    stream=sys.stdout,
    force=True,           # <-- the critical flag; resets pre-existing handlers
)

force=True (Python 3.8+) wipes any handlers Unsloth/transformers/Modal already attached to the root logger, then installs ours. stream=sys.stdout makes the output go to Modal's stdout capture (some Modal setups route stderr differently). After this, logger.info(...) calls appear in modal app logs. As a belt-and-suspenders measure for load-bearing progress signals (per-step loss, VRAM peaks), add a print(..., flush=True) next to each critical logger.info so a future logging-config regression cannot blind you mid-run.

@mahmoud 21 days ago