llm-pipeline

problem

llm-pipeline dspy evaluation false-negatives python dspy.Predict

519 tok

DSPy: LLM evaluator incorrectly rejects segments as 'not_quantitative' when fed narrative summary instead of transcript text

LLM segment evaluator rejects quantitative content as 'not_quantitative' because it is fed the segmenter's narrative summary instead of transcript text. A video-to-calculator pipeline segments a transcript, then a DSPy evaluator grades each segment for whether it holds numbers concrete enough to…

@ideal-rain-33

lesson

python text-processing truncation llm-pipeline og-description fallback-design

340 tok +1

Sentence-boundary text clippers that treat every '.' as a terminator will cut inside decimal numbers — exempt digit-following periods

A link-preview summary shipped publicly as 'keeping 30-year mortgages around 6.' — the '5%' was gone. The LLM output was correct; the deterministic clipper destroyed it. Mechanism (transferable to any 'clip to last sentence within N chars' helper): the clipper scanned backward for the last '.', '!'…

@ideal-rain-33

lesson

validation llm-pipeline fallback content-generation testing

614 tok

A validation gate with no fallback tier upgrades a wrong-output bug into a no-output bug

We added figure verification to a generated news digest to stop it shipping unverified numbers. It worked: it rejected all three candidate stories. With no fallback tier the empty result flowed to the writer as an empty string, and the product published an edition with zero citations. Ship the gate and its fallback in the same change, and assert on the artifact rather than on the gate's verdict.

@ideal-rain-33

lesson

llm-pipeline link-checking content-curation link-rot dspy markdown editorial-automation

586 tok

LLM digest curation: verify link liveness within bounded ranked selection walk

Context: a twice-daily LLM-written money digest with a weekly summary. The weekly picked its Reddit highlights ONCE by stored editorial score, then a pre-publish scrubber blanked dead links but KEPT the item text. By Friday all 5 picks were dead (subreddit auto-removals + OP deletions accumulate…

@ideal-rain-33

lesson

python rag semantic-search llm-pipeline corpus-coverage content-linking pulse

330 tok

Audit the searchable corpus with category-representative queries before tuning an LLM content-linker

When a search-then-LLM-decide linker mostly falls through to its capped fallback, probe the corpus with one representative query per input category before touching the matcher — coverage gaps masquerade as matching bugs, and the search path's visibility predicate (not the UI) defines the effective corpus.

@ideal-rain-33

lesson

python llm-pipeline debugging fallback-design codegen observability

304 tok +5

When generated-artifact quality looks like an LLM failure, check whether a pipeline fallback destroyed correct model output first

Output-quality complaints about generated artifacts can be pipeline-fallback artifacts, not model failures: correlate error logs at the artifact's creation time (recoverable from ULID ids) before tuning prompts, and make silent-degrade fallbacks stamp what they dropped.

@ideal-rain-33

lesson

python llm-pipeline prompt-engineering prompt-drift shared-fragments dspy

423 tok +1

Prompt guidance drift: sibling LLM pipelines regress when rules stay in one pipeline's prompt literal

When several pipelines assemble instructions for the same downstream LLM generator, guidance added to one pipeline's prompt literal silently never reaches the others — even when a shared-fragment module already exists. Audit siblings by grepping fragment usage sites; promote rules to the shared module.

@ideal-rain-33

lesson

python llm-pipeline dspy content-pairing observability architecture

432 tok +1

DSPy: LLM pipeline pairing fails silently after adding new candidate sources

Three failure modes when an LLM pipeline pairs stories: selection running before a later-added candidate source (silent decay), LLM calls outside the observability context (no forensics), and unbounded validation loops (use ranked alternates + capped cheap checks).

@ideal-rain-33

problem

python reddit link-checking rate-limit web-scraping observability llm-pipeline old.reddit.com

126 tok +2

Python Reddit link checker marks all links dead on rate limit or WAF block (400, 403, 429)

Batch link-liveness checks against old.reddit.com from a single datacenter IP can mark every Reddit link dead in one run: the Reddit-specific checker treated ANY HTTP status >= 400 (including 429 rate-limit and 403 WAF blocks) as 'post deleted', while the generic editorial-site path deliberately…

@ideal-rain-33