What a 58/58-green conversational eval suite does not tell you: five check blind spots and an over-cooperative simulator

Context

An LLM onboarding conversation has an eval harness: five scenarios, an LLM-driven simulated user, eleven to thirteen assertions per run. It reported 58/58 checks passed. Driving the same flow through a real browser in the same session surfaced defects in every category the suite claims to cover, and re-reading the green transcripts surfaced four more that the suite had walked straight past.

The suite was not badly written. Each blind spot is a reasonable check implemented one notch weaker than its name implies.

The five blind spots, each found inside a passing run

1. user_messages_persisted counts instead of validating alternation. One transcript had two consecutive User messages with no assistant turn between them, the second answering a question the assistant never asked. 13/13 passed, because four user messages were indeed present. This was the exact structural shape of a previously-fixed, separately-documented bug (the model authoring a turn in the user's voice at a resume boundary) — the fix landed, the regression check did not. Assert role alternation, not message count.

2. no_duplicate_content compares across messages, not within one. Passing run, verbatim: "Your account is all set, Mahmoud! You're all set, Mahmoud! Your default space is ready to go." Two near-identical clauses in one sentence, invisible to a pairwise cross-message similarity check.

3. no_implementation_leaks only inspected assistant messages. The date-picker widget posted 1990-06-15 (America/New_York) as the user's own chat message — raw ISO date plus raw IANA identifier, underscore included, attributed to a human. Every one of the five runs contained it. Leak checks must cover every role, because widgets author user-role content.

4. calculator_links asserted the link exists, not that any prose refers to it. A passing run ended: "…this is exactly where to start: Run through that and come back." Dangling colon; the recommended item is never named in prose at all, existing only as navigation metadata rendered into a button. "Run through that" has no antecedent. The link was present, so the check was satisfied.

5. One scenario's check set omitted the terminal assertion. The "user restarts the flow" scenario passed 12/12 while never collecting the user's name, never collecting their date of birth, and never emitting registration_complete. The scenario that best models a hesitant real user was the one that never proved they could finish. Every scenario whose name implies completion needs a completion assertion, or it is testing that the conversation doesn't crash.

The simulator is systematically easier than a human

Comparing five simulated runs against one real browser run:

• The simulator never produced a one-sentence answer. Every turn was 2-4 sentences of polished prose. The real user's first answer was nine words.
• The simulator never misunderstood a question. Asked "who are you looking out for — just yourself, or is there anyone else?", the real user replied "Not really, I just put whatever is left into savings" — a non-sequitur that forced a clarification turn. No simulated run ever triggered one, so the recovery path had zero coverage despite being exercised within four turns of real use.
• The simulator always answered completely in one turn. Real users drip-feed.
• All five runs shared one demographic (same age, family structure, income, occupation) because nobody passed the persona flag the harness already supported. Five runs, one user.
• The simulator only ever touches the chat. It never navigates away mid-conversation, never types while a response is streaming — the two actions that produced the session's most serious real defects (a modal trap, and messages being silently discarded).

Also: the report artifact truncated the thing the process required reading

The harness's own process doc says green checks are the floor and mandates a close read of at least one transcript per session. The transcript it writes truncates every message at ~200 characters with an ellipsis — and the truncated tail is precisely where the registration prompt, the recommendation transition, and tool results live. Two of the findings above were only visible after re-dumping the conversation from the database with a separate command. If your process says "read the artifact," the artifact must not elide.

Transferable rules

• Name your checks after what they assert, or make them assert what they're named. Every blind spot here is a gap between the check's name and its implementation, and the name is what people trust when they read a green run.
• A green suite plus a passing regression check for bug X is not evidence X can't recur; it's evidence X can't recur in the exact form the check inspects. Re-derive the bug's structural shape and assert on that.
• Budget one adversarial human pass per eval session and treat the simulator as a smoke test. Ours found, within twenty minutes: an input that silently eats text, a factual error in the first parroted fact, and two parallel UIs collecting the same three fields from the same user.
• If your harness supports persona variation, using the default persona for every run makes the run count misleading. Five identical users is n=1.
• Structured-input turns (button taps, widget submissions) are a distinct class from text turns and tend to escape prompt-level budgets and content rules. Ours drew a five-bubble, ~95-word reply to a submitted date.