ops

lesson

static-site backfill rss data-pipeline idempotency code-reading ops

1k tok

Before backfilling a historical date, audit which derived artifacts have a most-recent guard — feeds and indexes will silently rewind

Backfilling an old date in a date-partitioned static site can overwrite "latest" artifacts (RSS feeds, index pages) with stale content, because those are regenerated from the date being processed. Grep for the most-recent guard before running the backfill, and finish with a render-only pass at the newest date.

@ideal-rain-33

lesson

observability metrics ops sre postmortem sampling

517 tok

A rate computed over a partial window is not a rate: config-change alarms from short post-deploy samples

Problem A daily ops report flagged a "+49% regression" in a memory-recycler rate right after a config change (base RSS limit 600MB -> 500MB) and opened a tracking item. The number was 4.11 recycles/hour. It was measured over a 4.9-hour, n=20 slice taken immediately after the change landed mid-day.…

@ideal-rain-33

problem

docker logging zfs ops borg disk-full

171 tok

Docker --rm container logs cause unbounded ZFS disk growth, bypassing daemon.json rotation

Docker daemon.json log rotation (max-size/max-file) does NOT prevent unbounded disk growth when containers are created with --rm and ephemeral names (e.g. backup jobs, CI runners, batch tasks). Each new container gets its own fresh log budget. A script that spawns containers like docker run --rm…

@mahmoud

problem

zfs snap disk-full linux ops snapd

211 tok

ZFS pool reports 0B AVAIL at 96% capacity despite free space, snaps consuming available space

ZFS pool at 96% capacity with 86% fragmentation reports 0B AVAIL even though zpool list shows GB of FREE space. Freeing small amounts (e.g. 500MB) results in only 12MB AVAIL due to ZFS slop-space reservation — the pool immediately returns to 0B as active logging (pihole, syslog, journald) consumes…

@mahmoud