When generated-artifact quality looks like an LLM failure, check whether a pipeline fallback destroyed correct model output first

A generated financial calculator shipped with an identical, non-actionable call-to-action for every input case ('X is live.'). The obvious diagnosis — the model wrote a lazy static CTA, go tune the prompt — was wrong. Error logs timestamped at the artifact's creation minute showed the model had produced a proper conditional CTA (an IFS() formula with distinct per-case outputs); a downstream spreadsheet round-trip validator failed on unresolved cross-references inside the IFS comparisons, and the pipeline's retry/fallback path applied a flattened version with a constant string instead.

The transferable procedure:

1. Before tuning prompts over an output-quality complaint, pull pipeline error logs (Sentry etc.) filtered to the artifact's creation window — generation timestamps are usually recoverable from ids (ULIDs/KSUIDs encode time).
2. Distinguish three failure classes with different fixes: model produced bad output (prompt/model fix), validator falsely rejected good output (validator fix), fallback silently degraded output after a real failure (fix the root failure AND make the fallback loud — e.g. mark the artifact 'completed_with_issues' and surface that state where humans review quality).
3. Silent-degrade fallbacks are the worst of the three because they convert visible pipeline errors into invisible quality debt; every retry/fallback that ships a degraded artifact should stamp the artifact with what was lost.

In this case the 'lazy model' framing would have led to prompt churn with zero effect — the model was already doing the right thing.