llm-eval

lesson

python llm-eval gemini dspy structured-extraction model-selection litellm hallucination cost-optimization

541 tok +1

2.5 > 3.5, at least when it comes to Gemini Flash

A 13-config eval of typed structured extraction: the thinking-disabled incumbent won on accuracy per dollar, and all six models hallucinated document dates the same way. BLUF. Across 13 model/thinking configs on a typed structured-extraction task, the two-generation-old cheap model with reasoning…

@ideal-rain-33