PostgreSQL single-row UPDATE by primary key canceled by statement_timeout (psycopg2.errors.QueryCanceled: canceling statement due to statement timeout, CONTEXT: while updating tuple (N,M) in relation "user"). Confusing because the statement is trivially fast — a one-row PK update cannot be 'slow'. Setup: a daily cron tops up credit balances for system/service users; those same rows are decremented by every LLM call inside long-running background task transactions (multi-minute pipelines), making them hot rows.
statement_timeout counts LOCK-WAIT time, not just execution time. A one-row PK UPDATE that hits the timeout was blocked behind another transaction's row lock for the whole duration. The 'CONTEXT: while updating tuple (N,M)' line is the tell — the statement had already located its one tuple and was waiting for the lock on it.
The structural pattern to look for: shared 'system user' rows mutated inside long task transactions (credit/quota accounting decrements early in a multi-minute pipeline hold the row lock until commit) colliding with batch crons that sweep the same rows.
Fix for the sweeping cron: select ids with FOR UPDATE SKIP LOCKED and update only those — a locked hot row is by definition in active use and catches the next run:
locked_ids = session.scalars(
select(User.user_id)
.where(User.plan_type == plan, User.credits < threshold)
.with_for_update(skip_locked=True)
).all()Alternatives: SET LOCAL lock_timeout = '2s' + catch/skip per row, or move per-call accounting out of the long transaction (separate short transaction) so the hot row is never held for minutes. Diagnosing tip: the psycopg2 error's CONTEXT line plus checking what long-running jobs overlap the event timestamp (task-queue crons often pile up after deploys) identifies the lock holder class without pg_locks access.