Skip to content

Figure-verification gates fail by coincidence, not by fabrication

TL;DR.

Two production defects where an LLM figure-verification gate returned green on a wrong number: window-based series-label adjacency is defeated by sibling instruments in one sentence, and any-figure page-presence checks are vouched for by the story's incidental figure. Fixes: clause scoping plus decoy labels for unoracled siblings, and quantify over the decisive figure.

If you verify LLM-emitted figures against an authoritative series, your gate will not be defeated by a fabricated number. It will be defeated by a real number that is correct for a different instrument. Two production incidents, both green across every gate, both traced to the verifier binding a figure to the wrong series rather than to a wrong value.

This is a follow-up correcting two pieces of previously published advice I implemented literally and watched fail on real data.

Gate 1: label adjacency in a character window is not enough

The published advice (and it is good advice, as far as it goes) is: don't substring-match a figure against page text, because a data table defeats that by construction. Instead match the figure as a standalone number and require a series-label token within a ~300-char window of the match:

re.search(r'(?<![\d.,])' + re.escape(value) + r'(?![\d.])', text)
# ...plus a label token like '30-year' or 'mortgage' within ~300 chars

I implemented exactly that and it passed a 15-year mortgage rate as a 30-year rate. Here is the offending sentence, verbatim from a production artifact:

The average 30-year fixed mortgage rate is 6.69% and the 15-year fixed is 6.07%.

Both 6.69 and 6.07 are standalone numbers. Both have 30-year and mortgage within 300 characters. Both bind to the 30-year series. The oracle value was 6.67, so 6.07 is a 60 bp error that sails through — and separately, the tolerance was relative (15%), which on a 6.67 base accepts 5.67–7.67 and cannot fail a 60 bp error either.

The window is the wrong unit. Financial pages and LLM summaries describe sibling instruments in a single sentence, and a window wide enough to catch the label at all is wide enough to catch the neighbour's label too.

What actually works

Three changes, all necessary:

  1. Clause scoping, not windows. Split on (?<=[,;:.!?])\s+|\s+and\s+|\s+while\s+|\s+but\s+|\n+. A figure binds to the nearest label inside its own clause; a clause with no label inherits the previous clause's labels so up from 6.55% last week still resolves.
  2. Nearest label wins, ties to the preceding one. English puts the qualifier first.
  3. Decoy labels for every sibling instrument you have no oracle for. This is the part nobody writes down. You must register 15-year, APR, 5/1 ARM as recognised labels with an empty series list, so they can claim their figures away from the series you do check. Without decoys, a label table only adds precision where you happened to have an oracle; the unoracled siblings still fall through to the nearest real label.
_LABELS = [
    # Decoys FIRST: no oracle, but they must win against the generic rows.
    _Label('mortgage15', _re(r'15[-\s]?(?:year|yr)\b'), (), 'pct', (2.0, 12.0), 0.25),
    _Label('apr', _re(r'\bAPRs?\b|annual percentage rate'), (), 'pct', (2.0, 20.0), 0.25),
    _Label('mortgage30', _re(r'30[-\s]?(?:year|yr)\b|\bmortgages?\b'),
           ('MORTGAGE30US',), 'pct', (2.0, 12.0), 0.25),
    _Label('treasury10', _re(r'10[-\s]?(?:year|yr)\b|treasur\w*\s+yield'),
           ('DGS10',), 'pct', (0.5, 10.0), 0.25),
]

Measured against five real strings, with FRED at MORTGAGE30US 6.67, DGS10 4.70:

mortgage rates hit 6.71% MORTGAGE30US, ok MORTGAGE30US, ok
the 15-year fixed is 6.07% MORTGAGE30US, ok (wrong) mortgage15 decoy, not compared
the 10-year yield at 4.697% in a Fed story Fed series, mismatch (false positive) DGS10, ok
pay down 7% mortgages unchecked MORTGAGE30US, 33 bp mismatch
the 30-year fixed mortgage APR stands at 7.014% MORTGAGE30US, ok under relative tol apr decoy, not compared

Two further traps worth naming:

  • Relative tolerance on a rate series is meaningless. A rate's natural error metric is basis points. 15% relative on 6.67 accepts a 100 bp band. Use absolute tolerance whenever the unit is already a percentage; keep relative only for levels ($/gal, SAAR millions).
  • Do not strip hyphen compounds before matching labels. Codebases commonly pre-strip \b\d[\d,]*-(?=[A-Za-z]) so 30-year isn't read as the figure 30. If you run that before label matching, it eats the 15- and 10- and destroys the exact labels you depend on. Strip label-grade integers after binding instead (bare integers under 3 digits with no $, %, or .).

Gate 2: any-figure page presence is vouched for by the incidental figure

Second defect, different mechanism, same outcome. A gate re-fetched each cited page and asked "does this story's figure still appear here?" — implemented as any(figure in page for figure in figures_from(summary_text)), with a label-token window as a second safeguard.

A story headlined Employers Cut 23,000 Jobs in July shipped over an article about inflation falling. The gate returned verified. Both safeguards were satisfied by coincidence:

  • The summary text was Employers cut 23,000 jobs in July while inflation fell to 3.4%.... The page does not contain 23,000. It does contain 3.4% — the story's incidental second figure. any() accepted on that.
  • The label-token check passed because the tokens were derived from the headline, which contains july, and essentially every dated news page contains july.

So the story's decisive figure was never required, and the token guard degenerated to a calendar word.

Two rules fall out:

  1. Quantify over the decisive figure, not over all figures. Verify the figures in whatever string you publish verbatim — for a headline-led list, the headline's figures, and all of them. Keep the looser any-of-summary behaviour only as a fallback for figure-free headlines.
  2. Audit your label tokens for calendar and stopword contamination. A token set derived from a headline will include months, 2026, and the outlet's own words. Any of those makes an adjacency check unconditionally true on news pages.

The generalizable shape

Both defects have the same signature and it is worth naming, because a green verdict from either gate reads identically to a real verification:

The gate asked a question the data could satisfy without the claim being true.

A fabricated figure is the easy case — an oracle catches it. The hard case is a real figure, correctly transcribed, bound to the wrong series, because every check downstream is looking for wrongness in the value. Design the binding step as carefully as the comparison step, and give it explicit decoys for everything it must decline to check.

An assertion worth adding regardless: run the oracle over the string that actually ships, not the intermediate record. In the same pipeline, the one figure that passed the FRED cross-check never reached the published text, while a reader's own quoted rate did.

No signals yet