Setup
An LLM content pipeline generates, from one source segment, both a title and a decision string. A later step composes the page's meta description by prepending the title question to the decision, deduping so the same question is not stated twice:
if title.endswith('?') and normalize_spaced(title) not in normalize_spaced(decision):
parts.append(title)Eight unit tests passed, including a dedupe test that fed the identical question in both slots.
What shipped in the smoke test
Running the composer over 209 real production rows produced, on one view:
"Can I afford the Ferrari maintenance cost per year? Can I afford the true ongoing costs of a Ferrari?"
Both strings were generated from the same segment by the same model. They are near-paraphrases, not duplicates -- zero substring overlap after normalization, ~90% semantic overlap. The dedupe was structurally incapable of catching it, and the composed snippet read worse than the unfixed original it was meant to improve.
Why the unit test could not catch it
The test asserted the exact-duplicate case because that is the case a human invents when writing a dedupe test. The real distribution is paraphrase, and you cannot invent a representative paraphrase from imagination -- you have to read what the model actually emitted. One pass over production rows surfaced it in seconds; no amount of additional hand-written cases would have.
Rules
- Two fields generated from one source are correlated, not independent. Any dedupe, diff, or "is this new information" check between them must assume paraphrase, not repetition. Exact-substring and set-equality checks are the wrong instrument by construction.
- Prefer a structural predicate over a similarity threshold. The fix here was not fuzzy matching (unpredictable, needs a tuned cutoff nobody can defend in three months) but a rule about form: a decision that is already question-form is already a hook, so do not prepend the title question. It is one line, it is explainable, and it degrades safely -- worst case you skip an improvement, never emit a duplicate.
- Run any generated-text transform over a real corpus before shipping it, and read the output. Not a sample of three. The full set is usually one cheap API pagination away, and it is the only place the paraphrase distribution lives. Ours: 209 rows, ~7 seconds, one defect found that nine tests missed.
- Then add the regression test from the real row, verbatim. The invented case and the observed case are different tests and you want both.
The adjacent finding, same corpus pass
The same production sweep answered a question the codebase could not: 74 of 209 rows (35%) had the field under 60 characters, and only 45 were question-form. Those distribution facts justified the change far better than the single anecdote that started it -- and they came free with the smoke test.