A generated financial calculator shipped with an identical, non-actionable call-to-action for every input case ('X is live.'). The obvious diagnosis — the model wrote a lazy static CTA, go tune the prompt — was wrong. Error logs timestamped at the artifact's creation minute showed the model had produced a proper conditional CTA (an IFS() formula with distinct per-case outputs); a downstream spreadsheet round-trip validator failed on unresolved cross-references inside the IFS comparisons, and the pipeline's retry/fallback path applied a flattened version with a constant string instead.
The transferable procedure:
- Before tuning prompts over an output-quality complaint, pull pipeline error logs (Sentry etc.) filtered to the artifact's creation window — generation timestamps are usually recoverable from ids (ULIDs/KSUIDs encode time).
- Distinguish three failure classes with different fixes: model produced bad output (prompt/model fix), validator falsely rejected good output (validator fix), fallback silently degraded output after a real failure (fix the root failure AND make the fallback loud — e.g. mark the artifact 'completed_with_issues' and surface that state where humans review quality).
- Silent-degrade fallbacks are the worst of the three because they convert visible pipeline errors into invisible quality debt; every retry/fallback that ships a degraded artifact should stamp the artifact with what was lost.
In this case the 'lazy model' framing would have led to prompt churn with zero effect — the model was already doing the right thing.