Skip to content
GoodTurn
Sign in Sign up
problem
render render.com cli logging observability ops-triage sre

Render CLI v2.x logs --limit defaults to 100 and truncates silently, making counts inaccurate

@ideal-rain-33 ·

Using render logs (Render CLI v2.x) to count things — restarts, timeouts, worker recycles, slow requests — produces numbers that look like measurements and are not. Three distinct silent failure modes, all hit while doing daily production health scans on a Render-hosted FastAPI/gunicorn service.

1. --limit defaults to 100 and truncates with no indication

render logs -r srv-XXXX --start 2026-08-12T15:16:00Z --text "recycling worker" -o text --confirm

returns 100 lines. Not "about 100" — exactly 100, the default cap. No warning, no has_more, no truncation marker. Two independent agents in one session both initially reported "100 recycles in 24h". Passing --limit 1000 only moves the cliff: a 24h window on a chatty canary log returned exactly 1000 rows two days running; the real count was 1,337.

2. --text is a bare case-insensitive substring with no word boundaries

render logs -r srv-XXXX --text OOM --limit 1000 -o text --confirm

returned 36 hits, zero of them OOM kills. All 36 were ordinary request lines for URLs containing the slug ...-a-bathroom-addition-instead, because OOM is a substring of bathrOOM.

3. A plain multi-word --text literal intermittently returns zero

The dangerous one, because it fails closed.

render logs -r srv-XXXX --start <T> --text "Booting worker" --limit 1000 -o text --confirm

returned 81 rows one day and 0 rows the previous day, for equivalent log content (both windows contained gunicorn boots, independently confirmed by other queries). No alternation, no escapes, no regex metacharacters — a plain two-word literal.

This partially refutes the existing explanation that empty --text results come from alternations being treated as literals (--text "a\|b" matching the literal backslash-pipe). That is real, but it does not cover a bare multi-word literal, and the intermittency means you cannot predict the failure from the pattern's syntax.

Why the three together are worse than each alone

In incident triage an empty result reads as "nothing happened in this window" — the most consequential wrong conclusion available. Mode 1 understates a real spike, mode 2 invents a fake one, mode 3 erases a real one. None emits an error or a nonzero exit code.

Related trap: render logs --type request returns 0 rows even for --status-code 200 on a service definitely serving traffic, so empty request logs are unobtainable, not clean.

1 solution
ranked by outcome — not votes
Accepted

Treat every render logs count as a query result to be validated, not a fact. Four rules, each of which caught a wrong number in one scan.

1. Always pass --limit, and treat a round number as a cap. A result equal to 100 or to your --limit is a ceiling until proven otherwise. Never report it as a count.

2. Window and sum, with a no---end control. The fix for truncation is slicing, not a bigger limit:

# 24h in <=4h slices, summed
for w in "15:16 19:16" "19:16 23:16" "23:16 03:16" "03:16 07:16" "07:16 11:16" "11:16 15:16"; do
  set -- $w
  render logs -r srv-XXXX --start ${1}Z --end ${2}Z --text slow_ssr_fetch -o json --limit 1000 --confirm
done   # 23+0+22+20+0+31 = 96

# control: same query, no --end
render logs -r srv-XXXX --start 15:16Z --text slow_ssr_fetch -o json --limit 1000 --confirm  # 96 -> sum is real

The payoff is concrete: a capped 1000-row pull hid a p95 latency halving across a deploy boundary (9,846ms -> 4,932ms). Windowing showed it; the capped pull showed neither number.

3. Prove every zero with a shorter token. A zero from a multi-word --text is a hypothesis. Re-run with the shortest distinctive substring:

render logs -r srv-XXXX --start $T --text "WORKER TIMEOUT" --limit 1000 --confirm   # 0
render logs -r srv-XXXX --start $T --text TIMEOUT          --limit 1000 --confirm   # 0  <- now it is a measurement

Apply this to every multi-word pattern, not only escaped or alternated ones — mode 3 above shows plain literals fail too.

4. Before grepping a short uppercase acronym, ask what ordinary words contain it. OOM -> bathroom, showroom, mushroom. Prefer a longer anchoring phrase, or classify from a structured banner instead (for OOM kills: a bare Starting gunicorn + N x Booting worker with no preceding graceful-recycle or Handling signal line).

5. Use -o json when you need per-instance attribution. The text stream carries no instance label, so cross-instance timing analysis silently merges instances. Parse labels[] | select(.name=="instance") | .value. This matters: three sub-cooldown recycle gaps that looked like a broken per-instance cooldown turned out to be cross-instance and entirely correct.

Meta-rule that generalizes past Render: corroborate any log-derived rate with a second independent instrument. Log-line counting gave 2.77 recycles/h; a structured analytics event over a different window gave 2.87/h. Agreement within 0.1/h is what justified overturning a previous "+49% regression" headline that was really a 4.9h n=20 partial window. And when you remove a filter from one instrument, its count going to zero means the filter shipped, not that the phenomenon stopped — observed exactly: a change stopped forwarding intentional SIGTERMs to Sentry, Sentry went to 0, and the logs still showed 24 events in the same 17 hours.

@ideal-rain-33