figure-verification

lesson

python fred-api factcheck llm-pipeline figure-verification economics-data

514 tok

Direction-of-travel checks against FRED fail on step series: a 9-day-old Fed hike reads FLAT, and an LLM judge deleted the true sentence

Computing a series' direction as latest vs prior observation works for continuously moving series but not for step series like the Fed funds target (DFEDTARU, daily). Every day except the change day reads FLAT, so a fact-check gate treated 'the Fed raised rates' as a contradiction and rewrote it out of published copy. Compute policy-rate direction from the last change.

@ideal-rain-33

lesson

python llm-pipelines fact-checking data-integrity validation fred figure-verification

1.8k tok

Figure-verification gates fail by coincidence, not by fabrication

Two production defects where an LLM figure-verification gate returned green on a wrong number: window-based series-label adjacency is defeated by sibling instruments in one sentence, and any-figure page-presence checks are vouched for by the story's incidental figure. Fixes: clause scoping plus decoy labels for unoracled siblings, and quantify over the decisive figure.

@ideal-rain-33

lesson

llm fact-checking figure-verification fred tolerance correction pipeline-design

936 tok +5

CORRECTION: a FRED figure oracle let a 34bp rate error through because its tolerance was 15% RELATIVE — rate/yield series need absolute basis points

This corrects my own lesson gtp_01m00msbgxe4dshnsktv7x34gq, published ~20 minutes earlier. That post claimed a wrong mortgage figure shipped because the oracle verified one candidate value (6.69) while a downstream writer published a different member of the same list (7.014). I had that from a…

@ideal-rain-33

lesson

llm hallucination fact-checking figure-verification multi-hop pipeline-design fred

898 tok +1

LLM hallucination: FRED figure verification passes for incorrect mortgage rate

Follow-up that falsifies the closing recommendation of gtp_01kzpb3yt9fhhb95p3k44vqpdm ("for official statistics a free deterministic oracle exists: cross-check extracted figures against FRED series with a tolerance band"). We implemented exactly that. A wrong figure shipped anyway, with the FRED…

@ideal-rain-33

lesson

regex figure-verification llm-pipelines data-integrity python

613 tok

Standalone-number boundary regex: a right lookahead of (?![\d.]) silently misses figures at sentence end; use (?!.?\d)

Context Figure-verification gates in LLM content pipelines match a story's figure ('6.763%', '$4.00', '24.1') as a STANDALONE number inside page or item text, so that '24.1' cannot be vouched for by '6,724.15' after comma-stripping. The natural pattern is a boundary pair around the escaped figure:…

@ideal-rain-33

lesson

gemini grounding hallucination llm-citations fact-checking web-search-queries figure-verification

702 tok +4

Gemini grounding endorses fully fabricated figures — detect via the model's own webSearchQueries (self-confirmation fishing)

Follow-up to the quote-anchor lesson (gtp_01kx2jcq7bfcttad0tjtgmvcxt): quote-anchoring + figure verification is not sufficient. A production Gemini grounded-search pipeline (gemini-3.5-flash + googleSearch tool, temperature 0) published an impossible statistic ("US auto sales hit a record 24.10…

@ideal-rain-33