A common backstop for Postgres-backed job queues (BeanQueue/bq-style) is a cron that requeues orphaned work: WHERE state = 'PROCESSING' AND created_at < now() - interval '2 hours'. This predicate conflates queue wait with execution time. If the task table records only created_at (enqueue time) and no processing-started timestamp, a task that waited 2h in a backlog and then started running is indistinguishable from one orphaned by a dead worker. The sweep flips it back to PENDING while the original worker is still executing it, producing duplicate execution — double external side effects (sandbox spend, API calls, webhooks) and duplicate result rows when both invocations commit.
The failure window is exactly the queues most likely to have a sweep: single-threaded workers running hour-long tasks, where head-of-line blocking routinely pushes queue wait past the sweep threshold.
Mitigations, in order of preference:
A common backstop for Postgres-backed job queues (BeanQueue/bq-style) is a cron that requeues orphaned work: WHERE state = 'PROCESSING' AND created_at < now() - interval '2 hours'. This predicate conflates queue wait with execution time. If the task table records only created_at (enqueue time) and no processing-started timestamp, a task that waited 2h in a backlog and then started running is indistinguishable from one orphaned by a dead worker. The sweep flips it back to PENDING while the original worker is still executing it, producing duplicate execution — double external side effects (sandbox spend, API calls, webhooks) and duplicate result rows when both invocations commit.
The failure window is exactly the queues most likely to have a sweep: single-threaded workers running hour-long tasks, where head-of-line blocking routinely pushes queue wait past the sweep threshold.
Mitigations, in order of preference: