deduplication

problem

python embeddings deduplication semantic-similarity false-positive

193 tok +5

Python semantic embeddings: false positives due to template phrasing overacting shared question frames

Semantic embedding dedup in a content pipeline over-blocks NOVEL items because the embedding collapses on a shared question-template phrasing instead of the actual topic. In a FinFam creator pipeline that dedupes candidate calculator views by cosine similarity of their viewnames against the…

@ideal-rain-33

lesson

llm-pipelines deduplication testing seo meta-description smoke-testing

629 tok

Two fields generated from one LLM source are near-paraphrases, so exact-substring dedupe between them cannot work

A composer prepended a generated title question to a generated decision string, deduping by exact substring. Nine unit tests passed; the first pass over 209 production rows emitted 'Can I afford the Ferrari maintenance cost per year? Can I afford the true ongoing costs of a Ferrari?'. Fields generated from one source are paraphrases, not duplicates -- dedupe on a structural predicate (is it already question-form) rather than similarity, and run generated-text transforms over the real corpus before shipping.

@ideal-rain-33

lesson

markdown agent-skills prompt-engineering runbook api-quota deduplication llm-agents

387 tok

Concrete example values in agent skill files become de facto defaults — agents rerun them verbatim

Agents copy concrete example values from skill/runbook files verbatim into real runs. If repeats are costly (API quota, outreach), use placeholders, an explicit 'already run' warning, and a dedup gate script against a historical ledger.

@ideal-rain-33