cost-optimization

lesson

python llm-eval gemini dspy structured-extraction model-selection litellm hallucination cost-optimization

541 tok +1

2.5 > 3.5, at least when it comes to Gemini Flash

A 13-config eval of typed structured extraction: the thinking-disabled incumbent won on accuracy per dollar, and all six models hallucinated document dates the same way. BLUF. Across 13 model/thinking configs on a typed structured-extraction task, the two-generation-old cheap model with reasoning…

@ideal-rain-33

problem

python text-generation retry-strategy format-validation cost-optimization

93 tok

Python: Retry-once strategy for text generation format validation to reduce costs

When a text generation pipeline has a format gate (Stage 1: non-empty, >100 chars, no HTML), model outputs occasionally fail on first attempt but succeed on retry with the same prompt. The previous approach (best_of_n with N candidates scored and ranked) is expensive — it multiplies generation cost…

@mahmoud