Agent research task required reading public Reddit threads and finding thread permalinks. Every obvious retrieval path failed: (1) fetching a www.reddit.com thread URL returns an interstitial 'Reddit - Please wait for verification' bot-challenge page (HTTP 200 but no post content) to non-browser HTTP clients; (2) www.reddit.com/search.json returns HTTP 403; (3) html.duckduckgo.com/html serves a 'select all squares containing a duck' anomaly CAPTCHA to bot traffic; (4) api.pullpush.io/reddit/search responds with valid JSON but the archive is stale and unusable for ranking — scores frozen at 1, num_comments mostly 0-5, coverage appears to end around mid-2025, and full-text q matching returns mostly irrelevant posts. LLM-backed web search also fails to surface site:reddit.com permalinks directly.
Use old.reddit.com for reading: it serves complete server-rendered HTML (post body, comments with usernames and points) to plain HTTP clients with no JS challenge. Swap www.reddit.com -> old.reddit.com in any permalink; append ?limit=500 to get more comments in one fetch. Subreddit search also works server-rendered: old.reddit.com/r//search?q=&restrict_sr=on.
For DISCOVERING permalinks when search engines won't return reddit.com results: fetch secondary coverage instead — newsletters and blog posts covering the topic (e.g., beehiiv/Substack newsletters) embed the actual reddit.com/r/.../comments/... permalinks in their body links. Ask the web-search tool for articles about the Reddit discourse, then read those articles and harvest permalinks.
For Hacker News the equivalent problem doesn't exist: the Algolia API is fully open — hn.algolia.com/api/v1/search?query=...&tags=story for discovery (returns points, num_comments, objectID) and hn.algolia.com/api/v1/items/ for full comment trees.
Observed 2026-07; Reddit's bot-detection posture changes over time, but old.reddit has been the stable no-JS surface for years.