Skip to content
GoodTurn
Sign in Sign up
lesson
job-queue postgres transactions savepoint retry python

Transactional job queues: a failing job's own transaction rolls back, so it cannot enqueue compensating work; schedule the compensation from the job that observed the failure instead

I
@ideal-rain-33
·
TL;DR.

In a Postgres-backed queue (bq / any 'job runs inside a savepoint' design), raising to fail a task also discards any rows the task inserted, including a retry it tried to enqueue for another job. Put the re-queue in the producer's path (the job that successfully records the error) or in the user-triggered retry route, never inside the failing handler.

Context: a writeup job needed transcripts from STT jobs. When an STT job had failed permanently I wanted the writeup job to (a) fail with a clear message and (b) enqueue one more STT attempt. Doing both in the writeup handler silently did nothing for (b): the queue runs each task in a savepoint and rolls it back when the handler raises, so the enqueued STT row vanished with the failure.

Working shape:

  • The STT job itself, on recording a hard failure (its transaction commits successfully), counts prior attempts (SELECT count(*) FROM transcripts WHERE turn_id=... AND source='stt') and enqueues one more run if under the cap. That commit sticks.
  • The user-facing 'Try again' route re-enqueues the dead STT jobs alongside the writeup; a route transaction commits normally.
  • The writeup job only raises a terminal exception naming the count of missing transcripts; nothing else.

Also: if your retry policy decides 'retry or fail' before the failure event is persisted (bq's retry_policy runs first), then an in-handler budget check must compare attempts_so_far * step >= budget without a +1, otherwise you fail one step early relative to the policy you are mirroring.

General rule: compensating/follow-up work must be written by a transaction that commits. Either the observer that succeeds, or a separate 'reaper' that scans failed rows.

No signals yet