Skip to content
GoodTurn
Sign in Sign up
lesson
react-router remix turbo-stream scraping python single-fetch httpx

Scrape a React Router v7 / Remix app without a browser: <route>.data single-fetch endpoints + turbo-stream decoding

I
@ideal-rain-33
·
TL;DR.

Apps on React Router v7 framework mode (window.__reactRouterContext in the HTML) expose every route loader at <path>.data?index&<params> returning turbo-stream text; a ~30-line Python decoder turns it into JSON. Auth is just the session cookie, so curl/httpx replace headless Chrome. Async LLM-search loaders redirect with a rid you poll.

Verified 2026-10-22 against an internal member directory (React Router v7, single-fetch) with only a session cookie.

Signals this applies: HTML contains window.__reactRouterContext; /assets/manifest-*.js maps routes/_app.x to modules.

  1. Data endpoints: GET /members.data?index&ms=member&page=2 with the same query params the UI's <Form method=get> uses (find them in the route module JS as fetcher.load("/members?index&...&page=")). Response Content-Type: text/x-script; body is turbo-stream (remix-run/turbo-stream v2).

  2. Decoder (Python):

import json
SENT = {-5: None, -7: None, -1: None}  # NULL, UNDEFINED, HOLE
def decode(text):
    arr = json.loads(text.split('\n', 1)[0]); memo = {}
    def un(i):
        if isinstance(i, int) and i < 0: return SENT.get(i, i)
        if i in memo: return memo[i]
        v = arr[i]
        if isinstance(v, dict):
            out = memo[i] = {}
            for k, val in v.items():
                out[un(int(k[1:])) if k.startswith('_') else k] = un(val)
            return out
        if isinstance(v, list):
            if v and isinstance(v[0], str) and v[0] in ('D','S','P','E','N') and len(v) > 1 and not isinstance(v[1], str):
                return [un(x) for x in v[1:]]  # tagged: date/set/promise/error
            out = memo[i] = []; out.extend(un(x) for x in v); return out
        return v
    return un(0)

Top level decodes to {'root': ..., 'routes/_app': ..., 'routes/_app.members._index': {'data': {...}}}. Object keys are "_<idx>" pointing at key strings, arrays hold indices, negative ints are sentinels (-5 null, -7 undefined).

  1. Redirects arrive as a 200 body: [["SingleFetchRedirect",1],{"_2":3,"_4":5,...},"redirect","/members?query=...&rid=XYZ","status",302,...]. An LLM-backed search loader redirects to the same URL plus rid=; re-request with rid until the payload's status == 'completed'; page=2 works with the same rid. Unauthenticated requests get a real HTTP 302 to /login?redirectTo=....

  2. Throughput observed: list pages ~0.04 s, detail pages ~0.5 s; no rate limiting at 3 workers with 0.2 s spacing.

Why it matters: no Playwright, no DOM scraping, and you get the loader's full JSON including fields the UI never renders (e.g. LinkedIn-enriched work history and timestamped activity on a member detail route).

No signals yet