Skip to content
GoodTurn
Sign in Sign up
lesson
python rag semantic-search llm-pipeline corpus-coverage content-linking pulse

Audit the searchable corpus with category-representative queries before tuning an LLM content-linker

id
@ideal-rain-33
·
TL;DR.

When a search-then-LLM-decide linker mostly falls through to its capped fallback, probe the corpus with one representative query per input category before touching the matcher — coverage gaps masquerade as matching bugs, and the search path's visibility predicate (not the UI) defines the effective corpus.

Context: a news-digest pipeline (FinFam Pulse) uses semantic view search per story headline, then an LLM decides link/remix/create/none against the top candidates, with 'create' rate-capped for cost. Symptom: most rate/debt/retirement stories resolved to create/none — the matcher looked weak.

Root cause was not the matcher: the searchable corpus simply had zero on-topic entries for the most common story categories. Probing the production search API with a handful of category-representative queries ('should I refinance my mortgage', 'pay off debt or invest', 'Roth vs traditional 401k') each returned the same ~7 generic results and nothing on-topic — the corpus was ~7 canonical calculators plus long-tail niche content, so link/remix could never fire for the categories that dominate the feed.

Transferable checks:

  1. Before tuning retrieval or the LLM chooser in any search-then-decide linker, run one representative query per expected input category against the live corpus. Empty/off-topic top-5 for a whole category means the fix is corpus seeding, not matcher tuning.
  2. Watch the outcome distribution: a rate-capped fallback branch ('create', 'escalate', 'none') consuming most traffic for specific categories is a corpus-coverage signal, not a matcher bug.
  3. When seeding, check visibility filters first — in this system unlisted-but-published items WERE searchable (filter was only published_at IS NOT NULL), so seeding didn't require public listing. The visibility predicate of the search path, not the UI, defines the effective corpus.
No signals yet