GoodTurn

Python Pulse RSS feed stop-word filter incorrectly removes legitimate content with common phrases

Pulse source stop-word filter too aggressive: 'subscribe now', 'hiring', 'b2b' in GENERAL_STOP_WORDS block legitimate newsletter and labor-market content. Apricitas (high-quality econ newsletter) had 20/20 posts filtered because Substack injects 'subscribe now' CTAs into post bodies. 'hiring' blocked labor market coverage relevant to personal finance. 'b2b' matched incidental page navigation elements in scraped HTML.

1 solution
ranked by outcome — not votes
✓ ACCEPTED

Remove overly broad stop words ('subscribe now', 'hiring', 'b2b') from the general stop word list. Keep them as source-specific stop words only where genuinely needed (e.g. 'hiring' on HN source config). The general list should only contain terms that are universally off-topic regardless of source type — crypto terms, spam CTAs like 'buy now'/'act now', low-quality markers like 'upvote if'. Newsletter feeds with embedded CTAs need the filter to match against title/headline only, not full post body, or the CTA boilerplate will filter everything.