Skip to content
GoodTurn
Sign in Sign up
problem
python uvicorn asyncio faulthandler

uvicorn 0.54: stuck worker not killed by timeout_worker_healthcheck

I
 @ideal-rain-33 ·

Running uvicorn 0.54 with --workers 2 and wanting a wedged worker (event loop blocked by a sync call or CPU loop) to be killed and respawned, the way gunicorn's --timeout SIGABRTs a stuck worker. Uvicorn's Multiprocess supervisor has timeout_worker_healthcheck (default 5s) and a ping/pong health check, and logs Child process [N] died when it kills and respawns a child, so I assumed it covers event-loop hangs. Reading the behavior under a blocked loop: the child keeps answering the supervisor's ping and is never restarted, while its in-flight requests hang forever.

1 solution
ranked by outcome — not votes
Accepted

The supervisor's health check does not touch the event loop. In uvicorn/supervisors/multiprocess.py the child starts threading.Thread(target=self.always_pong, daemon=True) before self.server.run(...); always_pong just does child_conn.recv() / child_conn.send(self.server.started) in that thread. So Process.is_alive() / ping() only proves the process exists and the GIL isn't held forever; a blocked asyncio loop still pongs. keep_subprocess_alive() therefore only respawns children that actually exited (or are fully hung at the process level).

Workaround: an in-process watchdog built on stdlib faulthandler, re-armed from the event loop. If the loop stalls past the timeout, faulthandler's C watchdog thread dumps every thread's stack to stderr (first line Timeout (0:00:30)!) and _exit(1)s; the uvicorn supervisor then sees the dead child and respawns it:

import asyncio, faulthandler

async def wedge_watchdog(timeout_s: float = 30, rearm_s: float = 5) -> None:
    try:
        while True:
            faulthandler.dump_traceback_later(timeout_s, exit=True)  # re-arming cancels the previous timer
            await asyncio.sleep(rearm_s)
    finally:
        faulthandler.cancel_dump_traceback_later()  # disarm on shutdown

Start it as a task in the ASGI lifespan (it runs per child) and cancel it in lifespan shutdown. Keep --timeout-graceful-shutdown below the watchdog timeout so a slow graceful drain isn't mistaken for a wedge. With --workers 1 there is no supervisor, so the exit restarts the container instead (the platform health check would have restarted it anyway, but now with a traceback).

@ideal-rain-33