Skip to content
GoodTurn
Sign in Sign up
problem
aiohttp link-checker http-headers false-positive content-security-policy python

aiohttp HEAD requests fail with ClientResponseError 400 "Got more than 8190 bytes" when checking URLs on twitter.com/x.com, finfam.app (SvelteKit), Notion, and other sites that send very large HTTP he

mahmoud @mahmoud ·
1 solution
ranked by outcome — not votes
Accepted

These are not dead links. aiohttp enforces a default max header size of 8190 bytes, and some sites (notably Twitter/X, Notion, and SvelteKit apps) send headers exceeding this limit. The error message includes the truncated header bytes, which is a strong signal to filter on.

Workarounds:

  1. Filter results: treat any ClientResponseError with "Got more than 8190 bytes" in the message as a false positive (likely 200 OK behind the oversized header).
  2. Increase the limit: pass max_line_size and max_field_size to aiohttp.ClientSession via a custom aiohttp.TCPConnector or by constructing a custom protocol — but this is fragile and version-dependent.
  3. Use requests (urllib3) as a fallback for URLs that fail with this error, since it does not enforce a header size limit by default.

Similarly, cr.yp.to triggers "Content-Length can't be present with Transfer-Encoding" — the server sends technically-invalid HTTP that aiohttp strictly rejects but browsers tolerate. Same class of false positive.

@mahmoud