A retry loop around litellm.completion (gemini/gemini-2.x-flash with googleSearch grounding tool) retried only ONE flake mode — a successful response with empty groundingChunks — but let a second flake mode, transport exceptions, escape and kill the whole job with remaining attempts unused. Production sequence: attempt 1 returned grounded text but zero grounding chunks (known intermittent Gemini behavior) → retry fired; attempt 2 raised litellm.Timeout: Connection timed out after None seconds. ('None seconds' = litellm not propagating the timeout= kwarg into the Gemini transport error message) → exception propagated past the loop to the outer handler → entire feature returned empty. The wasted irony: attempt 1's grounded TEXT was perfectly usable — only the citation metadata was missing — but the exception path discarded it.
Treat 'returned empty' and 'raised' as the same flake and retry both: wrap the completion call per-attempt in try/except; on exception, retry remaining attempts. Second, degrade instead of dying: if a PRIOR attempt already produced usable text (just missing metadata), a later attempt's exception should proceed with that text rather than raise — downstream fallbacks (search-based link resolution) can compensate for the missing citations. Only raise when all attempts fail with nothing in hand:
for attempt in range(max_attempts):
try:
resp = litellm.completion(..., timeout=60)
except Exception as e:
if attempt == max_attempts - 1 and not grounded_text.strip():
raise # total failure, nothing usable
log(f'attempt {attempt+1} failed ({e}), ' + ('proceeding with prior text' if grounded_text.strip() else 'retrying...'))
if attempt < max_attempts - 1:
time.sleep(1.5)
continue
grounded_text = resp.choices[0].message.content or ''
... # parse metadata; break when chunks presentGeneral principle: when a retry loop guards a flaky external call, enumerate ALL observed flake modes (empty-but-successful AND raised) — a loop that retries only one mode silently converts the other into total failure. And before raising, check whether earlier attempts left partial results that downstream fallbacks can still use.