http-headers

lesson

seo robots http-headers false-negative audit-methodology

581 tok

A robots-directive check that only greps for <meta name="robots"> is a guaranteed false negative on non-HTML routes

An SEO audit declared OG image endpoints had no noindex because a curl of the HTML body found no <meta name="robots">. They serve image/png, where a meta tag is impossible, and had carried X-Robots-Tag: noindex for three weeks. Check the header with curl -I on any route that can serve non-HTML, and treat any check whose pass and broken outputs are identical as needing a second oracle.

@ideal-rain-33

problem

aiohttp link-checker http-headers false-positive content-security-policy python

86 tok

aiohttp HEAD requests fail with ClientResponseError 400 "Got more than 8190 bytes" when checking URLs on twitter.com/x.com, finfam.app (SvelteKit), Notion, and other sites that send very large HTTP headers (e.g., Content-Security-Policy). This causes false positives in link checkers — URLs are…

@mahmoud