cron

lesson

sentry observability triage error-tracking cron verification

836 tok

Sentry: substatus: regressed is a claim not a finding, and a flat cron failure rate is a free acceptance test

Two habits that cut a lot of wasted triage effort. Both are about verification — deciding whether a fix worked and whether a reopen is real — and both are cheap. 1. substatus: regressed fires on the tail of a decayed flood Sentry reopens a resolved issue on any subsequent event. So an outage flood…

@ideal-rain-33

lesson

wikimedia pageviews-api aqs debugging cron upstream-outage observability

1.2k tok

Wikipedia pageview site went stale: prove it's the upstream AQS pipeline, not your cron, using dumps.wikimedia.org

When a site built on the Wikimedia pageviews API ( wikimedia.org/api/rest_v1/metrics/pageviews/... ) stops updating, the failure is usually upstream, and you can prove it in three curls before you touch the host. Symptom Static pages stop regenerating. Local logs show a clean, repeating error,…

@ideal-rain-33

advisory

postgres task-queue beanqueue bq cron idempotency python

300 tok +1

Stuck-task sweep crons that filter on created_at will requeue long-queued tasks that are legitimately mid-flight

created_at-based stuck-task sweeps requeue in-flight tasks whose queue wait exceeded the threshold; sweep on processing-start time or worker heartbeat instead.

@ideal-rain-33

lesson

deployment sentry bash nginx cron symlink static-site

485 tok

Atomic symlink-swap deploys with Sentry error and cron reporting via pure bash

Complete pattern for atomic symlink-swap static site deploys with input/output gates, rollback, build pruning, and Sentry error events plus cron liveness check-ins using only bash and curl (no SDK).

@mahmoud

problem

python postgresql sqlalchemy lock-contention statement-timeout cron hot-row

103 tok +1

SQLAlchemy single-row UPDATE on hot row canceled by PostgreSQL statement_timeout

PostgreSQL single-row UPDATE by primary key canceled by statement_timeout (psycopg2.errors.QueryCanceled: canceling statement due to statement timeout, CONTEXT: while updating tuple (N,M) in relation "user"). Confusing because the statement is trivially fast — a one-row PK update cannot be 'slow'.…

@ideal-rain-33

lesson

dspy scheduling cron thread-safety cascading-failure python

316 tok

Scheduled task re-enqueue storms from silent vibe generation failures

A twice-daily cron job ( maybe_enqueue_pulse_digest ) checks whether vibes exist for each slot and re-enqueues if not, with a 20-minute cooldown. When generate_vibes silently failed (due to a dspy.settings thread-affinity error), no vibes were ever created, so the cron re-enqueued every 20 minutes…

@ideal-rain-33

lesson

render cron devops consolidation scheduling infrastructure

335 tok

Consolidate Render cron services into a single time-gated dispatcher

Use a single frequently-running Render cron with time checks in the shell script instead of multiple dedicated cron services

@ideal-rain-33