Three failure modes when an LLM pipeline pairs stories: selection running before a later-added candidate source (silent decay), LLM calls outside the observability context (no forensics), and unbounded validation loops (use ranked alternates + capped cheap checks).
A daily-brief pipeline pairs one macro news story with one micro human story (a Reddit post that humanizes the headline). The pairing kept failing despite an LLM selection step designed for exactly this. Three structural lessons from the root-cause:
1. Selection stages silently decay when a new upstream source is added. The micro-story selector originally ran against scraped macro headlines. Later, a grounded-search stage was added that usually produces the actual lead story — but it ran AFTER selection, so the selector never saw the story it was supposed to pair against. Nothing errored; the pairing just quietly stopped correlating. When adding a new candidate source to a pipeline, grep every downstream consumer of the old candidate list and re-check stage ordering.
2. Any LLM call outside your logging/callback context is invisible at the worst time. The selection call ran outside the DSPy callback context that captured all other calls into the debug archive, and its rationale output field was discarded. Result: the one stage whose judgment was being questioned was the only stage with zero forensic trail. Rule: every LLM call in a pipeline goes inside the same observability context, and every output field either lands in a trace or gets deleted from the signature.
3. Bounded ranked-alternates beat agentic validation loops. The chosen micro story's link must be live (posts get deleted; low-engagement posts are filtered). Instead of an agentic select->validate->reselect loop (unbounded latency in a pipeline that already runs ~10 min), have the LLM return 2x the needed ids in priority order, then walk them through a cheap validity check (HTTP liveness, ~1s each, capped at 12 checks) and take the first N live ones. One LLM call, deterministic worst case, and the discarded candidates + reasons go in the trace.
Also worth encoding in the selection prompt when pair quality is the objective: pair quality beats individual story size (a medium story with a perfect human echo beats the biggest story with none), with an explicit escape hatch flag for truly extraordinary news where relatability is sacrificed.