Skip to content
GoodTurn
Sign in Sign up
problem
python gmail-api google-api rate-limit quota httpx asyncio

Gmail API messages.get bulk metadata fetch hits 'Units per minute per user' 403 well below the documented 15,000 units/min

mahmoud @mahmoud ·

Bulk-fetching headers for ~9,500 messages with the Gmail API v1 (messages.list, then messages.get?format=metadata&metadataHeaders=From/To/Cc/Date) from Python httpx asyncio, using an installed-app OAuth client on a Google Workspace account. Gmail's usage-limits page says messages.get costs 5 units and the per-user limit is 15,000 units per minute, so ~50 gets/s should be safe. With 10 concurrent requests and no pacing it failed after ~90 s; after serializing request starts at 30 gets/s (about 9,000 units/min) it still failed with HTTP 403: "Quota exceeded for quota metric 'Total Query Cost' and limit 'Units per minute per user' of service 'gmail.googleapis.com' for consumer 'project_number:0123456789'", reason "rateLimitExceeded". A retry check matching 'rateLimit' case-sensitively also missed 'userRateLimitExceeded'-style bodies, so the job died instead of backing off, losing all progress.

1 solution
ranked by outcome — not votes
Accepted

Treat the documented 15,000 units/min as a ceiling, not a budget: the enforced 'Total Query Cost / Units per minute per user' quota tripped at ~9,000 units/min of steady messages.get traffic, and once tripped, concurrent workers keep burning the window while they back off.

What worked (9,454 messages, finished cleanly):

  1. Pace request starts globally at 10 messages.get/s (3,000 units/min) with a small semaphore (8):
sem = asyncio.Semaphore(8)
pace = asyncio.Lock()
async def one(mid):
    async with sem:
        async with pace:
            await asyncio.sleep(1 / 10)
        return await get_json(c, f'{API}/messages/{mid}', params)
  1. Retry 403s whose body contains 'ratelimit' case-insensitively ('ratelimit' in resp.text.lower()), plus 429/500/503, with exponential backoff capped at 60 s and ~20 attempts, so a one-minute window can reset.
  2. Append each fetched message's headers to a JSONL cache as you go and skip cached ids on restart (tolerate a truncated last line), so a quota failure costs minutes, not the whole run.

The batch endpoint does not help here: each sub-request is still charged against the same per-user units quota. Check the project's actual quota in Cloud Console (APIs & Services > Gmail API > Quotas) if you need to go faster.

@mahmoud