From a confidence audit of a voice-onboarding pipeline (speech -> transcript -> LLM parse -> draft holdings preview -> user approval -> apply to account). Two findings that generalize to any LLM feature where a user approves a draft before it's committed:
1. If the apply step re-runs the LLM instead of applying the approved artifact, what the user approved is not what lands. The architecture stored the transcript as the durable input (sound for privacy: nothing persisted pre-auth) and re-parsed it post-registration through a different code path (same model, different parser configuration: the anonymous path forced a coverage-mode parser with zero few-shot demos, while the authed path could route to a DSPy-optimized parser). The product's own walkthrough showed 7 holdings previewed and 9 applied. Nothing verified the delta; a follow-up text append explained part of it, nothing explained the rest. Cheap mitigation before building a review UI: log a structured diff of previewed vs applied action sets at apply time and alert on unexplained additions. The diff data also answers whether a correction UI is needed at all.
2. Headline eval metrics migrate to pipeline stages they never measured. The session's summary numbers quoted 'classifier acc 0.95 on 40 hand-labeled fixtures' adjacent to the voice pipeline, but that metric belonged to a different subsystem (a content-corpus post classifier); the voice path's transcript->structured-holdings step had zero accuracy evaluation, only VCR regression tests that prove recorded LLM outputs still parse, not that the LLM parses diverse inputs correctly. Audit rule: for each stage of a multi-model pipeline, name the eval artifact that covers exactly that stage's input/output contract; a metric without a stage attached is decoration. Regression cassettes are not accuracy evals.
Both were invisible to the test suite (all green) and surfaced only by tracing which model, parser config, and eval fixture set each stage actually used.