Python httpx 0.27: morningstar.com news content extraction fails despite 200 OK

The response is an AWS WAF Challenge action, and it is served as HTTP 202 Accepted with content-length: 0. Read the response headers and it is unambiguous:

HTTP/2 202
server: CloudFront
x-amzn-waf-action: challenge
content-length: 0
content-type: text/html; charset=UTF-8
x-cache: Error from cloudfront

202 is in the 2xx family, so every idiomatic success test passes it — resp.is_success (httpx), resp.ok (requests), raise_for_status(), curl -f, and any hand-rolled status_code < 400 guard. But the body is zero bytes, so every content-based check downstream silently degrades to "nothing found" rather than erroring. That combination — success status, empty body — is why the failure is invisible: a status-only checker says live, a body-only checker says dead/unverifiable, and neither says "blocked".

The reason nobody predicts this: the WAF behavior people special-case is 403 (and sometimes 405/429). Those are loud, and mature link checkers already fail open on 403 precisely because editorial sites WAF-block datacenter IPs. The Challenge action instead returns 202 so a real browser can transparently solve the challenge and retry; a programmatic client just gets an empty 202 and no signal at all unless it inspects headers.

The trap worth internalizing: making your fetcher look more browser-like can move you from a loud, correctly-handled 403 into a silent 2xx. Under curl, no-UA gave 403 and UA gave 202. The action selected is a function of the whole request fingerprint (TLS/HTTP2 profile, header ordering, egress IP reputation), not the UA string alone — which is why httpx returned 202 with and without the UA while curl's behavior differed. Choosing the action is also per-edge: the x-amz-cf-pop in the response varies, and the same URL can be clean from one egress IP and challenged from another, so this reproduces on a laptop while a server in a different region sees a normal 200 with full HTML. Do not conclude "not reproducible" from a single clean fetch elsewhere.

The fix is to stop treating liveness as a boolean and introduce a third unverifiable/blocked state, detected before any body inspection:

_WAF_CHALLENGE_HEADERS = ('x-amzn-waf-action',)  # add vendor equivalents as you meet them

def classify(resp) -> tuple[str, str]:
    # Blocked/challenged: a success status with no usable body.
    if any(h in resp.headers for h in _WAF_CHALLENGE_HEADERS):
        return 'unverifiable', f"waf_{resp.headers['x-amzn-waf-action']}"
    if resp.is_success and not (resp.text or '').strip():
        return 'unverifiable', f'empty_body_{resp.status_code}'
    if resp.status_code == 403:
        return 'unverifiable', 'fail_open_403'
    if resp.status_code >= 400:
        return 'dead', f'http_{resp.status_code}'
    return 'live', 'ok'

Two rules make this robust beyond this one vendor. (1) On any 2xx, assert the body is non-empty and long enough to contain what you intend to parse before you parse it — a 2xx with content-length: 0 is never a real article. (2) Never let unverifiable collapse into either live or dead: for a link checker, fail open but record the reason so the verdict is auditable later; for a content check such as "does this page contain the cited figure", unverifiable must NOT be reported as "figure absent", or a correct citation gets rejected because of a WAF, not because of its content. That second case is the expensive one — a verification gate that rejects on empty bodies will quietly discard good sources on every WAF'd domain.