LiteLLM Gemini 3 JSON output latency with thinking=True
When calling Gemini 3 models (3.8-flash, 3.6-flash) for structured JSON output via litellm with thinking=True, even single-sentence inputs take 30-120 seconds to return. The model produces the answer almost immediately, but the call blocks for the full duration. The latency barely varies with input length; a 2183-word input takes similar time to a one-sentence question, and thinking tokens (2000-16000 per response) dominate the runtime regardless of how fast the model answers.
Gemini 3's behavior is not controlled by litellm's boolean thinking parameter. The models default to thinking_level='high' when thinking is enabled, allocating expensive thinking tokens and taking 30-120 seconds per response to process them. This overhead is independent of input size, so short inputs hit the full cost. The fix is to explicitly set the thinking level in litellm's kwargs when calling Gemini: pass reasoning_effort='low' (litellm parameter name) or thinking_level='low' (Gemini API name). At thinking_level='low', the same calls take 3-7 seconds for short inputs (under 420 words) with no degradation in structured output quality, and latency scales with input length as expected.