Gemini API 429 RESOURCE_EXHAUSTED on googleSearch-grounded calls while plain calls succeed; error names no quota
Batch-scoring ~775 records with Gemini (gemini-3.8-flash via litellm/DSPy, tools=[{googleSearch:{}}], 2 grounded calls per record, ~1,500 grounded calls over ~30 min at 24 workers). Partway through, every call fails with 429 RESOURCE_EXHAUSTED: You exceeded your current quota, please check your plan and billing details. The error details carry only a Help link: no QuotaFailure, no quotaMetric, no RetryInfo, so you can't tell whether it's RPM, RPD, tokens or billing. Retrying a minute later with 4 workers still fails on every record, and litellm's num_retries doesn't help.
Find the exhausted quota by repeating the same request with and without the grounding tool. curl .../models/<model>:generateContent?key=$K -d '{"contents":[{"parts":[{"text":"ping"}]}],"tools":[{"googleSearch":{}}]}' returns the 429, while the same body without tools returns 200. So the Google Search grounding quota is exhausted, not the key, model RPM or billing. It doesn't recover within minutes, so treat it as a daily/windowed cap: stop the batch instead of retrying. Make the batch resumable (persist each result as it lands, and on rerun select only unscored records) and resume after the reset. Don't silently fall back to ungrounded calls if downstream logic relies on grounding sources (e.g. citation checks). Also log the full error body: truncating litellm exception text to ~300 chars cut it off before the details array. Check usage at https://ai.dev/rate-limit.