Skip to content
GoodTurn
Sign in Sign up
problem
python celery aws-ecs fargate django health-check docker

ECS Fargate healthCheck with celery -A app inspect ping restarts a healthy Django/Celery worker: Django import exceeds the 30 s timeout on 1 vCPU

I
 @ideal-rain-33 ·

Added an ECS container healthCheck to a Celery 5.6 prefork worker task on Fargate (1 vCPU / 4 GiB, Django 5 app, Redis broker): command celery -A core inspect ping -d "worker-prod@$(hostname)" --timeout 20 || exit 1, interval 120, timeout 30, retries 3, startPeriod 300. The worker started, drained its queue and logged nothing wrong, yet after startPeriod the task went UNHEALTHY and ECS logged (service ...) (task ...) failed container health checks and replaced it, every ~10 minutes. The same check on a 2 vCPU task of the same image looked fine. ecs execute-command was unavailable (TargetNotConnectedException, task role lacks SSM), so the health check command could not be timed in place. Expected inspect ping to be a cheap broker round trip.

1 solution
ranked by outcome — not votes
Accepted

Root cause: celery -A <django_app> inspect ping imports the Django app (settings, apps registry, and any startup secret fetch) before it can talk to the broker. On a Fargate task with 1 vCPU that import took ~40 s (measured from the worker's own startup log: secrets loaded to first settings-time log line = 34 s, banner at +38 s) and more while 4 prefork children compete for the same vCPU. ECS kills the check at timeout (max 60 s), counts it as a failure, and after retries consecutive failures marks the task UNHEALTHY; the service scheduler then stops it. On the 2 vCPU task the import took ~20 s, which is why it only failed on the small task. Raising timeout to 60 is marginal and flaps under load.

Fix: ping the worker from a bare Celery app that never imports Django. The pidbox control exchange is broker-level, so a Celery(broker=url) with no app module can control.ping a worker started by a different app, as long as broker URL and (if set) control_exchange / global_keyprefix match:

# scripts/celery_healthcheck.py  (run as: python scripts/celery_healthcheck.py worker-prod 10)
import socket, sys
from celery import Celery

prefix, reply_timeout = sys.argv[1], float(sys.argv[2])
app = Celery(broker=BROKER_URL)            # resolve from env / secret store, no Django
app.conf.broker_connection_retry = False
app.conf.broker_transport_options = {"socket_timeout": 10, "socket_connect_timeout": 10}
node = f"{prefix}@{socket.gethostname()}"  # matches --hostname=<prefix>@%h
replies = app.control.ping(destination=[node], timeout=reply_timeout)
sys.exit(0 if any(r.get(node, {}).get("ok") == "pong" for r in replies) else 1)

Task definition:

"healthCheck": {
  "command": ["CMD-SHELL", "python scripts/celery_healthcheck.py worker-prod 10"],
  "interval": 120, "timeout": 30, "retries": 3, "startPeriod": 300
}

Measured: 3.8 s for a pong (including a secret-store fetch for the broker URL), 15 s for a no-reply exit 1. Tasks at 1 and 2 vCPU all went HEALTHY after startPeriod. socket.gethostname() is what Celery substitutes for %h, so the destination matches on Fargate (ip-10-0-0-1.ec2.internal).

Two related gotchas on the same rollout: (1) if the ECS execution role lacks logs:CreateLogGroup, awslogs-create-group=true does not help a brand-new service; tasks fail with ResourceInitializationError: failed to validate logger args ... failed to create Cloudwatch log group until the group is created by hand. (2) Rolling a task def whose healthCheck references a script not yet in the deployed image starts the restart loop immediately; register an interim revision without healthCheck and let the CI build that ships the script apply the full definition.

@ideal-rain-33