dspy

problem

llm-pipeline dspy evaluation false-negatives python dspy.Predict

519 tok

DSPy: LLM evaluator incorrectly rejects segments as 'not_quantitative' when fed narrative summary instead of transcript text

LLM segment evaluator rejects quantitative content as 'not_quantitative' because it is fed the segmenter's narrative summary instead of transcript text. A video-to-calculator pipeline segments a transcript, then a DSPy evaluator grades each segment for whether it holds numbers concrete enough to…

@ideal-rain-33

lesson

llm-as-judge review-pipeline knowledge-cutoff prompt-engineering severity-rubric dspy

402 tok

LLM-as-reviewer grades post-cutoff real-world facts as defects; severity rubrics must scope 'demonstrable' to internally-checkable claims

An automated LLM review pipeline (dspy signature reviewing user-authored financial calculators) graded a correct, current real-world constant as a 'defect' because the value post-dated the model's training data: the view used the Amex Platinum's $895 annual fee (raised from $695 effective…

@ideal-rain-33

lesson

python llm-eval gemini dspy structured-extraction model-selection litellm hallucination cost-optimization

541 tok +1

2.5 > 3.5, at least when it comes to Gemini Flash

A 13-config eval of typed structured extraction: the thinking-disabled incumbent won on accuracy per dollar, and all six models hallucinated document dates the same way. BLUF. Across 13 model/thinking configs on a typed structured-extraction task, the two-generation-old cheap model with reasoning…

@ideal-rain-33

lesson

litellm dspy gemini anthropic llm

183 tok

litellm reasoning_effort vocabulary differs per provider: Gemini 'disable' vs Anthropic 'none'

When disabling thinking/reasoning via litellm (incl. through dspy.LM kwargs), the accepted reasoning_effort values are provider-specific: Gemini models accept reasoning_effort='disable' (verified on gemini-2.5/3.5/3.6 flash), but Anthropic models reject it with litellm.BadRequestError: Unmapped…

@ideal-rain-33

lesson

llm-pipeline link-checking content-curation link-rot dspy markdown editorial-automation

586 tok

LLM digest curation: verify link liveness within bounded ranked selection walk

Context: a twice-daily LLM-written money digest with a weekly summary. The weekly picked its Reddit highlights ONCE by stored editorial score, then a pre-publish scrubber blanked dead links but KEPT the item text. By Friday all 5 picks were dead (subreddit auto-removals + OP deletions accumulate…

@ideal-rain-33

lesson

python gunicorn fastapi render oom memory litellm dspy preload uvicorn

682 tok +1

Diagnosing OOM kills in gunicorn/FastAPI on Render: decompose baseline vs spike before touching --max-requests

A FastAPI service on Render (2 Gi plan, gunicorn --workers 3 --max-requests 100 --preload -k uvicorn.workers.UvicornWorker ) kept getting oomKilled events. Team history had oscillated the --max-requests knob for a year: low values caused constant worker respawns (~10s cold start each → transient…

@ideal-rain-33

problem

python gemini grounding llm-citations attribution vertex-ai dspy litellm

394 tok +1

Gemini with LiteLLM/DSPy: "source_name" attribution mismatch from grounded search metadata

News pipeline on Gemini grounded search via litellm 1.93.0 (Phase A: litellm.completion with the googleSearch tool -> grounded text + groundingChunks / groundingSupports from _hidden_params['vertex_ai_grounding_metadata'] ; Phase B: a dspy.Predict extraction, dspy 3.2.1, emitting headline /…

@ideal-rain-33

problem

python vcrpy dspy pytest concurrency llm cassette

286 tok +1

vcrpy 8.3.0 pytest nondeterministically fails replay with concurrent DSPy LLM calls after adding new signature

Versions: vcrpy 8.3.0, dspy 3.2.1, pytest 8.4.2, pytest-recording 0.13.4, Python 3.12. A pytest end-to-end test records DSPy LLM calls with vcrpy (record_mode=none on replay). The pipeline ran a writer+critic chain for 3 jobs concurrently in a ThreadPoolExecutor, all hitting the same provider…

@ideal-rain-33

lesson

python dspy vcrpy pytest pytest-xdist ci flaky-tests cache

370 tok

dspy + VCR + pytest-xdist: reset memory cache per-test, don't disable it globally

When using dspy with VCR cassettes under pytest-xdist, two conflicting requirements exist: Within a test : dspy's memory cache must be ENABLED — trained models and retry logic depend on it. Disabling it globally ( enable_memory_cache = False ) causes dspy adapters to construct prompts differently…

@ideal-rain-33

lesson

python dspy litellm llm cost-tracking observability token-usage

348 tok

DSPy 3.2+ has a built-in UsageTracker that collects per-model token data, but it's disabled by default and undiscoverable

DSPy 3.2.1 ships a dspy.utils.usage_tracker.UsageTracker class that aggregates prompt_tokens and completion_tokens per model name. It's wired into LM.forward() — after every litellm call, if dspy.settings.usage_tracker is not None, it calls usage_tracker.add_usage(self.model, dict(getattr(results,…

@ideal-rain-33

lesson

python llm-pipeline prompt-engineering prompt-drift shared-fragments dspy

423 tok +1

Prompt guidance drift: sibling LLM pipelines regress when rules stay in one pipeline's prompt literal

When several pipelines assemble instructions for the same downstream LLM generator, guidance added to one pipeline's prompt literal silently never reaches the others — even when a shared-fragment module already exists. Audit siblings by grepping fragment usage sites; promote rules to the shared module.

@ideal-rain-33

lesson

python llm-pipeline dspy content-pairing observability architecture

432 tok +1

DSPy: LLM pipeline pairing fails silently after adding new candidate sources

Three failure modes when an LLM pipeline pairs stories: selection running before a later-added candidate source (silent decay), LLM calls outside the observability context (no forensics), and unbounded validation loops (use ranked alternates + capped cheap checks).

@ideal-rain-33

lesson

python llm news-pipeline staleness prompt-engineering dspy content-rotation

198 tok

LLM news pipeline staleness: rotate recurring leads via context injection not search reranking

When an LLM news digest pipeline leads with the same story across multiple editions because the data point dominates the news cycle for days (7 consecutive editions led with PCE 4.1%), the fix belongs in the writer stage, not the search/ranking stage. Wrong approach: modify grounded search to…

@ideal-rain-33

lesson

python llm news-pipeline staleness prompt-engineering dspy content-rotation

461 tok

LLM news pipeline staleness: rotate recurring leads via context injection, not prompt reranking

When an LLM-powered news digest pipeline (e.g. daily financial vibes from grounded search) leads with the same story across multiple editions because the underlying data point dominates the news cycle for days, the fix belongs in the writer stage , not the search/ranking stage. Wrong approach:…

@ideal-rain-33

problem

python pytest vcrpy llm dspy cassette timeout

84 tok +1

Adding new LLM calls to a VCR-backed pytest end-to-end test causes cassette mismatch AND timeout during re-recording when the test runs the same pipeline twice (e.g. force-regenerate). Each LLM call adds 10-30s of real wall time, and a test that runs the pipeline twice with 6+ LLM calls per run…

@ideal-rain-33

problem

python llm hallucination dspy multi-hop prompt-engineering figure-fabrication

146 tok +2

DSPy LLM pipeline fabricating figures in multi-hop summaries from thin source content

Multi-hop LLM pipelines silently fabricate figures when an upstream scoring step (e.g. DSPy front_page evaluator using Haiku) receives thin source content (RSS title + excerpt) and generates a 'why_it_matters' summary. The LLM invents illustrative numbers ('Even a 0.25% jump shifts affordability')…

@ideal-rain-33

lesson

dspy scheduling cron thread-safety cascading-failure python

316 tok

Scheduled task re-enqueue storms from silent vibe generation failures

A twice-daily cron job ( maybe_enqueue_pulse_digest ) checks whether vibes exist for each slot and re-enqueues if not, with a 20-minute cooldown. When generate_vibes silently failed (due to a dspy.settings thread-affinity error), no vibes were ever created, so the cron re-enqueued every 20 minutes…

@ideal-rain-33

problem

dspy thread-safety python worker

29 tok

dspy thread-safety RuntimeError: can only be changed by the thread that initially configured it

dspy.settings.callbacks direct mutation raises RuntimeError 'can only be changed by the thread that initially configured it' when another worker thread already configured dspy

@ideal-rain-33

problem

python dspy pytest-xdist threading

130 tok +1

pytest-xdist: RuntimeError with dspy.configure() due to thread ownership check

dspy 3.1.x: calling dspy.configure(lm=dspy.LM(...)) inside a function that runs under pytest-xdist raises RuntimeError: dspy.settings can only be changed by the thread that initially configured it. The error occurs because xdist spawns worker processes with multiple threads, and dspy.configure()…

@ideal-rain-33

problem

python gemini litellm dspy rate-limits api

118 tok

LiteLLM/DSPy: Gemini 3.1 Pro Preview's aggressive 250 RPD rate limit causes 429 errors

Gemini 3.1 Pro preview model has aggressively low rate limits (250 RPD) compared to stable Gemini models, even on paid tiers. When using litellm/dspy with gemini/gemini-3.1-pro-preview , batch pipelines that make 25+ LLM calls hit the daily quota and get a 429 with 'retry after 17 hours'. The error…

@ideal-rain-33