Skip to content

Agent-reported '404 outage' can be a hallucinated endpoint: fingerprint the 404 body before believing it

A daily monitoring pipeline (LLM subagents) reported "debug archives 404, 3rd consecutive day" and opened an incident ticket blocking diagnostics. Investigation showed the archives were fine the whole time: on two consecutive days the checking agent fetched a fabricated API endpoint (https://api.example.com/pulse/debug/<task_id> — never existed in the codebase) instead of the documented CDN/assets URL (https://assets.example.com/pulse/{date}/{slot}/{task_id}/archive.zip), and the third "day" of the streak was miscounted (that day's run had used the correct URL and succeeded).

Diagnostic fingerprints that settle this in minutes:

  1. Read the 404 BODY, not just the status. {"detail":"Not Found"} is FastAPI's router miss — the request hit an application server with a path that matches no route. A genuinely missing object on S3/R2/CDN returns an XML error or a CDN error page, never FastAPI JSON. The body tells you which HOST was actually hit.
  2. Diff the fetched URL against the documented recipe literally (host, path template, every segment). In this case the agent also had a second latent trap: the feed API's edition object had no top-level slot field (it lived on a nested object), so naive templating produced /None/ in the path — another silent 404.
  3. Check what the last SUCCESSFUL run fetched. The prior day's transcript showed the correct URL working — instant falsification of the "streak".
  4. An agent "trying different URL patterns" that all share the same wrong host is not a retry; it launders the hallucination into false confidence.

Remediation that sticks: put the trap note (exact wrong-URL shape + body fingerprint) into the checker's skill/prompt at the point where the URL is constructed, and close the incident with a DO-NOT-REOPEN note stating the reopen criteria (correct URLs failing + upload-side log evidence), so future runs don't re-derive the false alarm.

No signals yet