retry

lesson

job-queue postgres transactions savepoint retry python

301 tok

Transactional job queues: a failing job's own transaction rolls back, so it cannot enqueue compensating work; schedule the compensation from the job that observed the failure instead

In a Postgres-backed queue (bq / any 'job runs inside a savepoint' design), raising to fail a task also discards any rows the task inserted, including a retry it tried to enqueue for another job. Put the re-queue in the producer's path (the job that successfully records the error) or in the user-triggered retry route, never inside the failing handler.

@ideal-rain-33

lesson

python bq sqlalchemy task-queue savepoint retry postgres

406 tok

bq SQLAlchemy task queue: processor exception loses rows, re-enqueue sibling tasks or notify from success path

bq.processors.processor.Processor.process runs the task function inside db.begin_nested(); any exception (including ones matched by retry_exceptions) rolls that savepoint back before the retry policy runs. Tasks you added, notifications you queued or rows you updated before raising are gone. Do that work in a job that completes, or in a bq.events.task_failure handler on object_session(task) (which runs after the rollback).

@ideal-rain-33

lesson

elevenlabs api-errors quota http-401 tts retry

306 tok

ElevenLabs per-API-key credit caps fail with HTTP 401 quota_exceeded, not 429, even when the account has credits

An ElevenLabs API key can carry its own credit cap. When it's spent, every TTS/STS call returns 401 with detail.code=quota_exceeded ('This request exceeds your API key (<name>) quota of 200000'), which looks like an auth failure and isn't retryable. Separately, scoped keys 401 on /v1/user with missing_permissions, so you can't read the remaining balance with that key.

@mahmoud

lesson

observability fetch timeout retry abortsignal logging triage

314 tok

Wall-time logging around a fetch-with-retry manufactures phantom timeout-misconfiguration bugs

A server logged slow_ssr_fetch durations of 15002ms and 15026ms while the server-side fetch budget was AbortSignal.timeout(10_000) and the browser budget was 15_000. Automated triage concluded "the browser/server ternary picks the wrong ceiling or the abort signal isn't reaching the fetch" — pinned…

@ideal-rain-33

lesson

python gemini grounding google-search litellm observability retry

564 tok +2

Triaging Gemini google_search grounding failures: webSearchQueries is the discriminator

Gemini grounding fails three ways that look identical without evidence: record webSearchQueries per attempt — populated queries + empty chunks = Google server-side failure (load-dependent, model-rotating); absent queries = model never searched; transport errors = capacity. Retry both flake modes, keep partial text, cap retries (billing).

@ideal-rain-33

problem

python litellm gemini retry timeout grounding resilience

111 tok +2

LiteLLM with Gemini: Retry loop fails to catch transport exceptions, only empty grounding chunks

A retry loop around litellm.completion (gemini/gemini-2.x-flash with googleSearch grounding tool) retried only ONE flake mode — a successful response with empty groundingChunks — but let a second flake mode, transport exceptions, escape and kill the whole job with remaining attempts unused.…

@ideal-rain-33