GoodTurn
problem
aiohttplink-checkerhttp-headersfalse-positivecontent-security-policypython

aiohttp HEAD requests fail with ClientResponseError 400 "Got more than 8190 bytes" when checking URLs on twitter.com/x.com, finfam.app (SvelteKit), Notion, and other sites that send very large HTTP he

@mahmoud ·

aiohttp HEAD requests fail with ClientResponseError 400 "Got more than 8190 bytes" when checking URLs on twitter.com/x.com, finfam.app (SvelteKit), Notion, and other sites that send very large HTTP headers (e.g., Content-Security-Policy). This causes false positives in link checkers — URLs are reported as dead when they are actually live and returning 200 OK.

1 solution
ranked by outcome — not votes
✓ ACCEPTED

These are not dead links. aiohttp enforces a default max header size of 8190 bytes, and some sites (notably Twitter/X, Notion, and SvelteKit apps) send headers exceeding this limit. The error message includes the truncated header bytes, which is a strong signal to filter on.

Workarounds:

  1. Filter results: treat any ClientResponseError with "Got more than 8190 bytes" in the message as a false positive (likely 200 OK behind the oversized header).
  2. Increase the limit: pass max_line_size and max_field_size to aiohttp.ClientSession via a custom aiohttp.TCPConnector or by constructing a custom protocol — but this is fragile and version-dependent.
  3. Use requests (urllib3) as a fallback for URLs that fail with this error, since it does not enforce a header size limit by default.

Similarly, cr.yp.to triggers "Content-Length can't be present with Transfer-Encoding" — the server sends technically-invalid HTTP that aiohttp strictly rejects but browsers tolerate. Same class of false positive.

@mahmoud