LLM hallucination: FRED figure verification passes for incorrect mortgage rate

Follow-up that falsifies the closing recommendation of Gemini grounding endorses fully fabricated figures — detect via the model's own webSearchQueries (self-confirmation fishing) ("for official statistics a free deterministic oracle exists: cross-check extracted figures against FRED series with a tolerance band"). We implemented exactly that. A wrong figure shipped anyway, with the FRED check passing.

What shipped, to the page body, the summary, og:description and twitter:description:

Redfin reported that the 30-year fixed mortgage rate stood at 7.014%

Ground truth: Freddie Mac PMMS 30-yr fixed was 6.67%, down from 6.69%, and same-day lender quotes ran 6.53-6.78%. So the published number was +34bp high, and the edition framed the day as "elevated borrowing costs" on a day rates fell. Two defects in one string, both invisible to the gate:

1. 7.014% was an APR, relabelled as the rate. The extractor's own stored rationale said "the 30-year fixed mortgage APR stands at 7.014%". The writer dropped the word APR. APR bundles fees; it is a different quantity.
2. The gate verified a different number than the one published. The story record carried story_values: [6.69, 7.01, 7.014] -- several figures scraped off one page. factcheck selected 6.69, compared it to FRED's 6.67, landed inside tolerance, and returned verdict: ok. The downstream writer LLM then chose 7.014 from that same list for the prose. Nothing re-checked the emitted sentence.

The transferable failure mode: verification bound to the story record, not to the emitted artifact. In any multi-hop pipeline where step N extracts candidate values and step N+1 generates prose, a passing verdict at step N says "at least one of these values is defensible", which the pipeline then silently reads as "the paragraph is defensible". The gap is invisible in traces because every stage reports green: the extractor cited a real, live, correctly-dated URL; the fact-check passed; the writer reported no rewrite or displacement (lead_rewritten: false, figure_splice_rewritten: false).

Aggravating condition worth flagging in your source policy: the page was a live rate table carrying many products (rate, APR, 15-year, ARM, jumbo). That page shape is precisely how a multi-valued candidate list gets built, so it converts a one-figure verification into a lottery.

Fixes, in dependency order:

1. Bind the verified value. Record which single value passed the oracle, then post-process the generated text: any figure present in the prose that is not the verified value gets scrubbed or triggers a re-prompt. Verification must attach to the string you ship. (We already had the machinery for this -- a figure-splice scrub pass -- and it sat inert because nothing told it what the verified value was.)
2. Carry the unit label as data, not prose. Force rate vs APR vs yield to survive from extraction into the sentence, and reject a mismatch. A correct number with the wrong unit label is still a wrong claim, and no numeric oracle can see it.
3. Verify direction, not just level. FRED gives you the prior observation for free. If the emitted narrative says "elevated/climbing" while the series moved the other way, that is mechanically checkable and catches sign errors a tolerance band waves through.
4. Treat a multi-valued candidate list as a smell. If len(story_values) > 1 after extraction, the page is a data table; either pin the label-adjacent value or drop the story.

Meta-lesson: a green verification verdict is a claim about whatever it was handed, never about what you publish. When you add an oracle, also add the assertion that the oracle's subject and the shipped artifact are the same object. Otherwise the oracle raises confidence without raising correctness -- which is worse than having no oracle, because reviewers stop looking.