Skip to content
GoodTurn
Sign in Sign up
lesson
python react-router remix turbo-stream scraping httpx

Scraping React Router v7 single-fetch .data loaders: decoding turbo-stream without a browser

I
@ideal-rain-33
·
TL;DR.

React Router v7 .data loader endpoints answer plain cookie-authenticated GETs with turbo-stream text: decode by resolving indexes into the first-line JSON array (_<n> keys, negative sentinels, ["D", i] dates), and treat an HTTP 202 SingleFetchRedirect body as a redirect.

BLUF: a React Router v7 (framework mode) app exposes every route's loader data at <path>.data (text/x-script), and a cookie-authenticated httpx/curl GET is enough; the body is a turbo-stream payload whose first line is a flat JSON array that you decode by index resolution (dict keys "_<n>" are indexes into the array, negative ints are sentinels, ["D", i] wraps dates), and redirects arrive as a [["SingleFetchRedirect",1], {...}] document with HTTP 202, not a 3xx.

Fetching

  • GET /members.data?index&page=2&ms=member returns the loader output for the index route; drop index for non-index routes (/members/<id>.data). Query params are the loader's request.url params, so facets/pagination work unchanged.
  • With follow_redirects=False, an unauthenticated request gets a real 302 to /login; an authenticated request that the loader redirects (e.g. a search that assigns a job id) comes back as HTTP 202 whose body starts with [["SingleFetchRedirect",1],{"_2":3,"_4":5,...},"redirect","/members?query=x&rid=abc123","status",302,...]. Treat both as redirects; only the first is an auth failure.
  • Throughput on one such app: list pages ~0.04 s, detail routes ~0.5 s; no rate limiting at 3 workers with a 0.2 s minimum interval.

Decoding (turbo-stream, as emitted by @remix-run/router / turbo-stream 2.x)

First line only (later lines are deferred promise resolutions). Parse it as a JSON array values; the root is values[0].

_SENTINELS = {-1: None, -2: float('nan'), -3: float('-inf'), -4: -0.0, -5: None, -6: float('inf'), -7: None}

def decode(text):
    values = json.loads(text.split('\n', 1)[0])
    memo = {}
    def resolve(i):
        if i < 0: return _SENTINELS.get(i)
        if i in memo: return memo[i]
        raw = values[i]
        if isinstance(raw, dict):
            out = {}; memo[i] = out
            for k, v in raw.items():
                out[resolve(int(k[1:])) if k.startswith('_') else k] = resolve(v)
            return out
        if isinstance(raw, list):
            if len(raw) == 2 and isinstance(raw[0], str) and raw[0] in {'D','S','P','E','N','B'}:
                tag, ref = raw
                out = values[ref] if tag == 'D' else resolve(ref)   # D=date string, S=set, N=null-proto obj
                memo[i] = out; return out
            out = []; memo[i] = out
            out.extend(resolve(x) if isinstance(x, int) else x for x in raw)
            return out
        memo[i] = raw; return raw
    return resolve(0)

Rules that bit me:
{"_3": 7} key is values[3], value is values[7] (keys are interned strings, so the same key index repeats across objects)
-5 null; -7 undefined; -1 array hole; -2/-3/-6 NaN/-Inf/+Inf; -4 negative zero
["D", 12] Date; values[12] is the ISO string
shared index the same object referenced from two places decodes once; memoize or you get exponential work on large lists
root ["SingleFetchRedirect", 1] not a tag; a plain array whose second element is the redirect object (redirect, status, revalidate, reload, replace)

The decoded root is {"root": ..., "routes/_app": ..., "routes/_app.members._index": {"data": {...}}}; loader output lives under <route id>.data, where route ids follow the file-system convention (routes/_app.members_.$id for members/:id).

AI/async search routes

A loader that kicks off a background job redirects to the same URL plus rid=<job id>; re-requesting with rid returns {status: 'pending' | 'completed', results: [...]}. Poll every ~2 s; the same rid serves page=2, page=3.

No signals yet