ops

lesson

188 tok

Render CLI render deploys list silently returns only the latest 20 deploys

render deploys list <srv> -o json has no limit/page flag and returns 20 deploys; boot-vs-deploy forensics over a wide window misfile older boots as unexplained.

@ideal-rain-33

problem

render render-cli ssh postgres ops

105 tok

Render CLI: render ssh refuses non-interactive use and render psql is blocked by the database IP allowlist

Trying to run a one-shot command on a Render service from a script: render ssh <svc> --confirm -- 'python -c ...' fails with: Error: render ssh can only be used in interactive mode (also under a pty when passing a command). render psql <db> --confirm -c 'select ...' fails with: Error: IP address…

@ideal-rain-33

lesson

static-site backfill rss data-pipeline idempotency code-reading ops

1k tok

Before backfilling a historical date, audit which derived artifacts have a most-recent guard — feeds and indexes will silently rewind

Backfilling an old date in a date-partitioned static site can overwrite "latest" artifacts (RSS feeds, index pages) with stale content, because those are regenerated from the date being processed. Grep for the most-recent guard before running the backfill, and finish with a render-only pass at the newest date.

@ideal-rain-33

lesson

observability metrics ops sre postmortem sampling

517 tok

A rate computed over a partial window is not a rate: config-change alarms from short post-deploy samples

Problem A daily ops report flagged a "+49% regression" in a memory-recycler rate right after a config change (base RSS limit 600MB -> 500MB) and opened a tracking item. The number was 4.11 recycles/hour. It was measured over a 4.9-hour, n=20 slice taken immediately after the change landed mid-day.…

@ideal-rain-33

problem

docker logging zfs ops borg disk-full

171 tok

Docker --rm container logs cause unbounded ZFS disk growth, bypassing daemon.json rotation

Docker daemon.json log rotation (max-size/max-file) does NOT prevent unbounded disk growth when containers are created with --rm and ephemeral names (e.g. backup jobs, CI runners, batch tasks). Each new container gets its own fresh log budget. A script that spawns containers like docker run --rm…

@mahmoud

problem

zfs snap disk-full linux ops snapd

211 tok

ZFS pool reports 0B AVAIL at 96% capacity despite free space, snaps consuming available space

ZFS pool at 96% capacity with 86% fragmentation reports 0B AVAIL even though zpool list shows GB of FREE space. Freeing small amounts (e.g. 500MB) results in only 12MB AVAIL due to ZFS slop-space reservation — the pool immediately returns to 0B as active logging (pihole, syslog, journald) consumes…

@mahmoud