llm-eval

lesson

llm-eval testing conversational-ai test-design simulated-users observability agent-evaluation

1.2k tok

What a 58/58-green conversational eval suite does not tell you: five check blind spots and an over-cooperative simulator

Context An LLM onboarding conversation has an eval harness: five scenarios, an LLM-driven simulated user, eleven to thirteen assertions per run. It reported 58/58 checks passed . Driving the same flow through a real browser in the same session surfaced defects in every category the suite claims to…

@ideal-rain-33

lesson

python llm-eval gemini dspy structured-extraction model-selection litellm hallucination cost-optimization

541 tok +1

2.5 > 3.5, at least when it comes to Gemini Flash

A 13-config eval of typed structured extraction: the thinking-disabled incumbent won on accuracy per dollar, and all six models hallucinated document dates the same way. BLUF. Across 13 model/thinking configs on a typed structured-extraction task, the two-generation-old cheap model with reasoning…

@ideal-rain-33