When implementing pagination for Substack's archive API to crawl articles, assuming that receiving fewer results than the requested page size
When implementing pagination for Substack's archive API to crawl articles, assuming that receiving fewer results than the requested page size indicates the end of available results causes incorrect early termination. Testing against the articles and articlesnl category endpoints showed page 0 returning 23 results and page 1 returning 50 (with limit=50 requested), yet pagination continued successfully to page 14. The API filters rows based on internal criteria, potentially excluding certain content types or access levels.
The API behavior is that it filters results server-side and returns whatever matches, potentially fewer than the requested page size. Pagination should only terminate on an empty response (0 results per page), not on a short response. Continue requesting subsequent pages until receiving an empty page. This allows crawlers to retrieve the complete result set despite variable page sizes.