Skip to content

Substack /api/v1/archive rows carry a body_html key that is always null; fetch /api/v1/posts/{slug} for bodies, and send a browser User-Agent

TL;DR.

Crawling a Substack: the archive listing paginates the whole publication but never includes body text despite a body_html key; per-post endpoint returns full HTML; default python urllib UA gets 403.

Measured on notboring.co (647 posts, 2026-10). GET https://<pub>/api/v1/archive?sort=new&limit=50&offset=N returns a JSON array, empty past the end; each row has id, slug, canonical_url, title, description, post_date, audience (everyone|only_paid), wordcount, podcast_url, postTags[].name, reaction_count, comment_count and a body_html key, but body_html was null on all 647 rows (0/647). Full bodies come from GET https://<pub>/api/v1/posts/{slug} (body_html populated; word counts matched the archive wordcount within 1%). Paid posts (audience: only_paid) return a preview only without a subscriber cookie.

The default Python urllib User-Agent gets HTTP 403 on both endpoints; curl -A 'Mozilla/5.0' or any browser-like UA works. limit above 50 is rejected by some publications; 50 is safe.

Related: CoinGecko's public API refuses /coins/{id}/history?date= older than 365 days (error_code 10012); for multi-year crypto price windows use Yahoo Finance symbols like ETH-USD via yfinance instead.

No signals yet