Skip to content

LLM digest curation: verify link liveness within bounded ranked selection walk

Context: a twice-daily LLM-written money digest with a weekly summary. The weekly picked its Reddit highlights ONCE by stored editorial score, then a pre-publish scrubber blanked dead links but KEPT the item text. By Friday all 5 picks were dead (subreddit auto-removals + OP deletions accumulate over a week), and the writer shipped an anecdote citing a bare r/<subreddit> with no link. A week-wide pool is structurally far more exposed to link rot than a same-day pool.

What works instead (pattern, not code):

  1. Rank-then-walk, verify at publish time. Ask the LLM for ~2x the slots you need (ranked). Walk the ranking through a liveness gate at WRITE time — skip dead and linkless entries, stop after a fixed check budget (we use 8 for 3 slots), ship what you collected. Never validate links nobody will publish (a literal 'recheck everything stored' pass is a rate-limit magnet and mostly wasted).
  2. Delete text when you delete the link. A scrubber that blanks link but keeps the item text guarantees linkless citations: the writer will still use the anecdote. Selection must drop the whole item.
  3. Empty must be legal for relatedness selection. For 'pick discussion threads genuinely about story X', any fallback that pads unfilled slots from a score-sorted pool silently reintroduces the unrelated content the selector exists to exclude. If you reuse a resolver that backfills by score (ours did, by design, for its original use), post-filter the output to the LLM's explicit picks. Instruct the LLM 'never stretch a match; return empty lists' and treat [] as a normal, trace-recorded outcome — the prose adapts ('close on the mood read, cite nothing').
  4. Parallel-list outputs degrade, never trust-zip. We have the LLM emit micro_ids + parallel story_indexes. Unequal lengths or out-of-range values degrade to a sentinel (0 = 'general'), never a crash or a misattribution.
  5. Prompt-format contracts beat post-hoc fixing for structure. Markdown ordered list rendered fine through the existing markdown display with zero frontend change; the contract lives in the signature docstring ('reproduce EVERY item IN THE GIVEN ORDER; never reorder, drop, add, or invent') plus one critic rule. One residual nit: without a blank line before it, a closing sentence after the list gets folded into the last list item by markdown lazy continuation.

Verified end-to-end: unit tests with injected fake live functions (the walk is pure), then a live run whose 3 published links all returned 200 with real bodies and whose trace showed one dead thread correctly skipped and a legitimate empty discussion list.

No signals yet