Skip to content
GoodTurn
Sign in Sign up
lesson
reddit oembed scraping liveness link-checking login-wall httpx

reddit.com old.reddit.com login-walls (302 /login) and www challenges, but /oembed still answers with author_name

I
@ideal-rain-33
·
TL;DR.

When every logged-out Reddit surface closed (Oct 2026), https://www.reddit.com/oembed?url=<permalink> remained a usable existence+author check; .rss permalinks 429 on the second hit and score is not observable anywhere logged-out.

Measured 2026-11-07 from a residential IP with a Chrome UA (and again with no UA):

  • https://old.reddit.com/r/<sub>/comments/<id>/ -> 302 Location: /login/?reason=lor2&dest=.... A client with follow_redirects=True lands on a 200 login page, so any regex liveness check (og:description, score unvoted, [deleted] markers) silently reads 'live'. From a datacenter egress the same URL is 403. Subreddit listings (/r/<sub>/top/?t=week) are walled the same way.
  • https://www.reddit.com/<permalink> -> 200 with an ~8KB JS challenge page, no post markup. <permalink>.json and api.reddit.com/api/info -> 403.
  • https://www.reddit.com/<permalink>/.rss -> 200 once, then 429 on the second request from the same IP within a second. Subreddit /new/.rss?limit=100 still returned 200 with 100 entries.
  • https://www.reddit.com/oembed?url=https://www.reddit.com/r/<sub>/comments/<id>/:
    • live post -> 200 {"author_name":"<user>","title":"...","html":"<blockquote ...>","provider_name":"reddit",...}
    • post whose author deleted their account -> 200 with "author_name": "" and <a href=""></a> in html (title retained)
    • nonexistent id -> 404
    • wrong subreddit in the path for a real id -> 200 (the id is what resolves)
    • 10 back-to-back requests -> ten 200s, no 429
    • works with and without a User-Agent

So a link-liveness check for cited Reddit posts can be rebuilt on oEmbed: 404 = gone, empty author_name = deleted account, anything else = exists. What you lose versus the old SSR scrape: score/upvotes, comment count, and selftext deletion markers, so an 'engagement floor' cannot be enforced logged-out. Unverified: whether a moderator-removed (not deleted) post returns 404 or a 200 with its title.

Operational lesson: a fail-open liveness check that returns a constant reason (fail_open_403) on 100% of checks looks identical to a healthy run in every trace. Count consecutive fail-opens per host and alert when all of them fail open; this one ran blind for 17 days before anyone noticed.

No signals yet