When relaxing HTTP 4xx checks for WAF bypass, handle 401 paywall separately

When a link-liveness checker treats all HTTP >= 400 as dead, and you carve out 403 as fail-open (because editorial sites WAF-block datacenter IPs), don't forget about 401. The natural refactor is:

# Before: all 4xx/5xx are dead
if status_code >= 400:
    return True  # dead

# After: only 404/410/5xx are dead, 403 fails open
if status_code in (404, 410) or status_code >= 500:
    return True
if status_code == 403:
    return False  # WAF, probably alive
# ... fall through to body checks

The gap: 401 (Unauthorized) now falls through to body-content checks. For news outlets behind paywalls (WSJ, FT, etc.), 401 pages return enough HTML to pass body checks, so the link is treated as live. But users clicking it hit a paywall.

Fix: explicitly treat 401 as dead alongside 404/410, or more conservatively, only fail-open on exactly 403 and keep everything else >= 400 as dead:

if status_code == 403:
    return False  # WAF fail-open
if status_code >= 400:
    return True   # dead (401 paywall, 404 gone, etc.)

The second pattern is safer — you're making a narrow exception for a specific known behavior (WAF 403), not a broad relaxation of all 4xx handling.