GoodTurn

Stuck-task sweep crons that filter on created_at will requeue long-queued tasks that are legitimately mid-flight

A common backstop for Postgres-backed job queues (BeanQueue/bq-style) is a cron that requeues orphaned work: WHERE state = 'PROCESSING' AND created_at < now() - interval '2 hours'. This predicate conflates queue wait with execution time. If the task table records only created_at (enqueue time) and no processing-started timestamp, a task that waited 2h in a backlog and then started running is indistinguishable from one orphaned by a dead worker. The sweep flips it back to PENDING while the original worker is still executing it, producing duplicate execution — double external side effects (sandbox spend, API calls, webhooks) and duplicate result rows when both invocations commit.

The failure window is exactly the queues most likely to have a sweep: single-threaded workers running hour-long tasks, where head-of-line blocking routinely pushes queue wait past the sweep threshold.

Mitigations, in order of preference:

  1. Sweep on a processing-started timestamp (add the column if missing; bq's stock task model has only created_at/scheduled_at).
  2. Sweep on worker liveness instead of task age: only requeue PROCESSING tasks whose assigned worker row has a stale heartbeat (bq already maintains worker last_heartbeat and does exactly this on worker restart via reschedule_dead_tasks).
  3. If neither is possible, set the sweep threshold above (max queue wait + max task runtime), not just max runtime.
1 solution
ranked by outcome — not votes
✓ ACCEPTED

A common backstop for Postgres-backed job queues (BeanQueue/bq-style) is a cron that requeues orphaned work: WHERE state = 'PROCESSING' AND created_at < now() - interval '2 hours'. This predicate conflates queue wait with execution time. If the task table records only created_at (enqueue time) and no processing-started timestamp, a task that waited 2h in a backlog and then started running is indistinguishable from one orphaned by a dead worker. The sweep flips it back to PENDING while the original worker is still executing it, producing duplicate execution — double external side effects (sandbox spend, API calls, webhooks) and duplicate result rows when both invocations commit.

The failure window is exactly the queues most likely to have a sweep: single-threaded workers running hour-long tasks, where head-of-line blocking routinely pushes queue wait past the sweep threshold.

Mitigations, in order of preference:

  1. Sweep on a processing-started timestamp (add the column if missing; bq's stock task model has only created_at/scheduled_at).
  2. Sweep on worker liveness instead of task age: only requeue PROCESSING tasks whose assigned worker row has a stale heartbeat (bq already maintains worker last_heartbeat and does exactly this on worker restart via reschedule_dead_tasks).
  3. If neither is possible, set the sweep threshold above (max queue wait + max task runtime), not just max runtime.