If you verify LLM-emitted figures against an authoritative series, your gate will not be defeated by a fabricated number. It will be defeated by a real number that is correct for a different instrument. Two production incidents, both green across every gate, both traced to the verifier binding a figure to the wrong series rather than to a wrong value.
This is a follow-up correcting two pieces of previously published advice I implemented literally and watched fail on real data.
Gate 1: label adjacency in a character window is not enough
The published advice (and it is good advice, as far as it goes) is: don't substring-match a figure against page text, because a data table defeats that by construction. Instead match the figure as a standalone number and require a series-label token within a ~300-char window of the match:
re.search(r'(?<![\d.,])' + re.escape(value) + r'(?![\d.])', text)
# ...plus a label token like '30-year' or 'mortgage' within ~300 charsI implemented exactly that and it passed a 15-year mortgage rate as a 30-year rate. Here is the offending sentence, verbatim from a production artifact:
The average 30-year fixed mortgage rate is 6.69% and the 15-year fixed is 6.07%.Both 6.69 and 6.07 are standalone numbers. Both have 30-year and mortgage within 300 characters. Both bind to the 30-year series. The oracle value was 6.67, so 6.07 is a 60 bp error that sails through — and separately, the tolerance was relative (15%), which on a 6.67 base accepts 5.67–7.67 and cannot fail a 60 bp error either.
The window is the wrong unit. Financial pages and LLM summaries describe sibling instruments in a single sentence, and a window wide enough to catch the label at all is wide enough to catch the neighbour's label too.
What actually works
Three changes, all necessary:
- Clause scoping, not windows. Split on
(?<=[,;:.!?])\s+|\s+and\s+|\s+while\s+|\s+but\s+|\n+. A figure binds to the nearest label inside its own clause; a clause with no label inherits the previous clause's labels soup from 6.55% last weekstill resolves. - Nearest label wins, ties to the preceding one. English puts the qualifier first.
- Decoy labels for every sibling instrument you have no oracle for. This is the part nobody writes down. You must register
15-year,APR,5/1 ARMas recognised labels with an empty series list, so they can claim their figures away from the series you do check. Without decoys, a label table only adds precision where you happened to have an oracle; the unoracled siblings still fall through to the nearest real label.
_LABELS = [
# Decoys FIRST: no oracle, but they must win against the generic rows.
_Label('mortgage15', _re(r'15[-\s]?(?:year|yr)\b'), (), 'pct', (2.0, 12.0), 0.25),
_Label('apr', _re(r'\bAPRs?\b|annual percentage rate'), (), 'pct', (2.0, 20.0), 0.25),
_Label('mortgage30', _re(r'30[-\s]?(?:year|yr)\b|\bmortgages?\b'),
('MORTGAGE30US',), 'pct', (2.0, 12.0), 0.25),
_Label('treasury10', _re(r'10[-\s]?(?:year|yr)\b|treasur\w*\s+yield'),
('DGS10',), 'pct', (0.5, 10.0), 0.25),
]Measured against five real strings, with FRED at MORTGAGE30US 6.67, DGS10 4.70:
mortgage rates hit 6.71% | MORTGAGE30US, ok | MORTGAGE30US, ok |
the 15-year fixed is 6.07% | MORTGAGE30US, ok (wrong) | mortgage15 decoy, not compared |
the 10-year yield at 4.697% in a Fed story | Fed series, mismatch (false positive) | DGS10, ok |
pay down 7% mortgages | unchecked | MORTGAGE30US, 33 bp mismatch |
the 30-year fixed mortgage APR stands at 7.014% | MORTGAGE30US, ok under relative tol | apr decoy, not compared |
Two further traps worth naming:
- Relative tolerance on a rate series is meaningless. A rate's natural error metric is basis points. 15% relative on 6.67 accepts a 100 bp band. Use absolute tolerance whenever the unit is already a percentage; keep relative only for levels ($/gal, SAAR millions).
- Do not strip hyphen compounds before matching labels. Codebases commonly pre-strip
\b\d[\d,]*-(?=[A-Za-z])so30-yearisn't read as the figure30. If you run that before label matching, it eats the15-and10-and destroys the exact labels you depend on. Strip label-grade integers after binding instead (bare integers under 3 digits with no$,%, or.).
Gate 2: any-figure page presence is vouched for by the incidental figure
Second defect, different mechanism, same outcome. A gate re-fetched each cited page and asked "does this story's figure still appear here?" — implemented as any(figure in page for figure in figures_from(summary_text)), with a label-token window as a second safeguard.
A story headlined Employers Cut 23,000 Jobs in July shipped over an article about inflation falling. The gate returned verified. Both safeguards were satisfied by coincidence:
- The summary text was
Employers cut 23,000 jobs in July while inflation fell to 3.4%.... The page does not contain 23,000. It does contain 3.4% — the story's incidental second figure.any()accepted on that. - The label-token check passed because the tokens were derived from the headline, which contains
july, and essentially every dated news page containsjuly.
So the story's decisive figure was never required, and the token guard degenerated to a calendar word.
Two rules fall out:
- Quantify over the decisive figure, not over all figures. Verify the figures in whatever string you publish verbatim — for a headline-led list, the headline's figures, and all of them. Keep the looser any-of-summary behaviour only as a fallback for figure-free headlines.
- Audit your label tokens for calendar and stopword contamination. A token set derived from a headline will include months,
2026, and the outlet's own words. Any of those makes an adjacency check unconditionally true on news pages.
The generalizable shape
Both defects have the same signature and it is worth naming, because a green verdict from either gate reads identically to a real verification:
The gate asked a question the data could satisfy without the claim being true.
A fabricated figure is the easy case — an oracle catches it. The hard case is a real figure, correctly transcribed, bound to the wrong series, because every check downstream is looking for wrongness in the value. Design the binding step as carefully as the comparison step, and give it explicit decoys for everything it must decline to check.
An assertion worth adding regardless: run the oracle over the string that actually ships, not the intermediate record. In the same pipeline, the one figure that passed the FRED cross-check never reached the published text, while a reader's own quoted rate did.