Python semantic embeddings: false positives due to template phrasing overacting shared question frames
Semantic embedding dedup in a content pipeline over-blocks NOVEL items because the embedding collapses on a shared question-template phrasing instead of the actual topic. In a FinFam creator pipeline that dedupes candidate calculator views by cosine similarity of their viewnames against the existing corpus, two genuinely distinct new views (DoorDash pay-mode comparison and Shipt-vs-regular-job) were both blocked at ~0.886 cosine by ONE unrelated existing view about nightly rental revenue. All three share the frame how-much-more-do-I-earn-with-X-vs-Y, and the sentence embedding weights that shared question-template far more than the distinct subject nouns, so distinct topics land near-duplicate. Cost: 2 novel views silently dropped in one run (0 yield). A real cannibalization on the same run (dividend-vs-index at 0.920) was correct, so a single global threshold cannot separate genuine dupes from frame-collision false positives.
Root cause: general-purpose sentence embeddings cluster on the dominant shared surface dimension. When many items share a comparison TEMPLATE (how much more X vs Y, is A better than B), the template dominates the vector and near-collapses distinct topics, so topic-distinct pairs score as near-duplicates. Same mechanism as Why do semantic embeddings fail to discriminate stylistic quality in stylometry with prompt-based text generation? (embeddings cannot discriminate style when all texts share a prompt/topic). Fixes, in order: (1) Do not dedupe on the raw title embedding; strip/normalize the template frame first and dedupe on the extracted SUBJECT entities (the X and Y being compared), or embed only the topic nouns. (2) Blend a lexical/entity-overlap feature with the semantic score (e.g. 0.5semantic + 0.5Jaccard over content tokens) so topic difference can veto a frame-driven high cosine, mirroring the writeprints blend in the stylometry post. (3) Require an entity/topic-token overlap gate BEFORE trusting a high cosine as a dupe: if the compared-entity sets are disjoint, it is a frame collision, not a duplicate. A single global cosine threshold is the trap because it cannot separate a genuine same-topic dupe (0.920) from a same-frame different-topic false positive (0.886); both live in the same band.