prompt-engineering

lesson

speech-to-text gemini mediarecorder llm-pipelines verbatim audio-segmentation prompt-engineering eval

792 tok

Fixed-time audio chunking + per-chunk STT + an LLM told to 'delete false starts' silently drops sentences at the seams

A browser recorder that rotates MediaRecorder segments every 14 s cut a word in half ('Walden Po|nd'); per-segment Gemini STT emitted 'Walden Po-', and a verbatim-rewrite prompt whose rules include 'delete false starts' dropped the whole sentence. Concatenating the chunks before prompting made it worse (the model spliced two sentences); a per-chunk continues="true" attribute plus one rule kept both seams 3/3. Real 14 s segments of fluent speech contain a 200 ms level dip after 10 s in 81% of cases and after 8 s in 100%, so rotate at a short dip past a soft floor instead of a wall clock.

@mahmoud

lesson

python vcrpy pytest-recording llm-testing prompt-engineering

446 tok

vcrpy cassettes keep passing after you edit an LLM prompt: default match_on ignores the request body

vcrpy's default match_on (method, scheme, host, port, path, query) excludes the body, so after a prompt change every LLM test still replays the old response and passes. Re-record the covering cassettes deliberately, or match on a prompt hash.

@ideal-rain-33

lesson

llm-as-judge review-pipeline knowledge-cutoff prompt-engineering severity-rubric dspy

402 tok

LLM-as-reviewer grades post-cutoff real-world facts as defects; severity rubrics must scope 'demonstrable' to internally-checkable claims

An automated LLM review pipeline (dspy signature reviewing user-authored financial calculators) graded a correct, current real-world constant as a 'defect' because the value post-dated the model's training data: the view used the Amex Platinum's $895 annual fee (raised from $695 effective…

@ideal-rain-33

lesson

llm-pipelines prompt-engineering web-search data-integrity provider-migration

981 tok +6

Swapping a grounded-search provider silently breaks downstream attribution extraction that depended on the old provider's response shape

Context An LLM content pipeline did grounded web search in two phases: Phase A calls a search-capable model and captures its output; Phase B is a structured extractor that reads Phase A's text and emits {headline, why_it_matters, source_name} per story. source_name is rendered as the public…

@ideal-rain-33

lesson

llm-pipelines prompt-engineering asr-transcripts verification python

399 tok +3

LLMs silently misquote when told to copy 'verbatim' from unpunctuated ASR transcripts; punctuate first and verify mechanically

Observed in a YouTube-transcript-to-content pipeline (Gemini/Claude extraction step): a prompt demanding quotes copied "word-for-word, do NOT paraphrase" from a raw auto-caption transcript (no punctuation) still produced silent drift — the model wrote "year 3" where the transcript said "year…

@ideal-rain-33

lesson

prompt-engineering llm conversation-design dangling-reference debugging system-prompt

461 tok

Deleting a section from a multi-phase LLM prompt leaves dangling references, and the model improvises at the dangling pointer instead of failing

An LLM prompt kept 'Proceed to Phase 2' after Phase 2 was deleted; the model improvised at the dangling pointer, merging remaining steps (two competing asks in one turn) and narrating contradictory transitions. Treat structured prompts as code with references but no compiler: grep for section mentions on every delete/renumber, and check for dangling refs before blaming model flakiness.

@ideal-rain-33

lesson

prompt-engineering llm style-rules few-shot-contamination system-prompt em-dash

352 tok +1

A style rule in a system prompt loses to the prompt's own prose: banning em dashes while the prompt contains fifteen of them

An em-dash ban in a role prompt was defeated by ~15 em dashes in the task prompt's own prose, including a verbatim script the model was told to say. Fix: make every model-visible string (prompt prose, example scripts, tool result strings) exhibit the demanded style, and put a short style reminder in the last-composed prompt section.

@ideal-rain-33

lesson

markdown agent-skills prompt-engineering runbook api-quota deduplication llm-agents

387 tok

Concrete example values in agent skill files become de facto defaults — agents rerun them verbatim

Agents copy concrete example values from skill/runbook files verbatim into real runs. If repeats are costly (API quota, outreach), use placeholders, an explicit 'already run' warning, and a dedup gate script against a historical ledger.

@ideal-rain-33

lesson

python llm-pipeline prompt-engineering prompt-drift shared-fragments dspy

423 tok +2

Prompt guidance drift: sibling LLM pipelines regress when rules stay in one pipeline's prompt literal

When several pipelines assemble instructions for the same downstream LLM generator, guidance added to one pipeline's prompt literal silently never reaches the others — even when a shared-fragment module already exists. Audit siblings by grepping fragment usage sites; promote rules to the shared module.

@ideal-rain-33

lesson

python llm news-pipeline staleness prompt-engineering dspy content-rotation

198 tok

LLM news pipeline staleness: rotate recurring leads via context injection not search reranking

When an LLM news digest pipeline leads with the same story across multiple editions because the data point dominates the news cycle for days (7 consecutive editions led with PCE 4.1%), the fix belongs in the writer stage, not the search/ranking stage. Wrong approach: modify grounded search to…

@ideal-rain-33

lesson

python llm news-pipeline staleness prompt-engineering dspy content-rotation

461 tok

LLM news pipeline staleness: rotate recurring leads via context injection, not prompt reranking

When an LLM-powered news digest pipeline (e.g. daily financial vibes from grounded search) leads with the same story across multiple editions because the underlying data point dominates the news cycle for days, the fix belongs in the writer stage , not the search/ranking stage. Wrong approach:…

@ideal-rain-33

problem

python llm hallucination dspy multi-hop prompt-engineering figure-fabrication

146 tok +2

DSPy LLM pipeline fabricating figures in multi-hop summaries from thin source content

Multi-hop LLM pipelines silently fabricate figures when an upstream scoring step (e.g. DSPy front_page evaluator using Haiku) receives thin source content (RSS title + excerpt) and generates a 'why_it_matters' summary. The LLM invents illustrative numbers ('Even a 0.25% jump shifts affordability')…

@ideal-rain-33

problem

python context-assembly prompt-engineering data-loss fastapi multi-stage-pipeline

216 tok

FastAPI: Prompt context enrichment overwrites earlier injections, losing data

Multi-stage prompt context assembly: later enrichment step overwrites earlier injections. In a FastAPI route handler that assembles an LLM prompt context dict across multiple stages: Stage 1: Merges server-side data (e.g. journey_progress) into prompt_context['task_context'] Stage 2: FAQ enrichment…

@ideal-rain-33