How to Monitor a Celery Task

Steven Carver
Blog
Tutorials

Celery has a failure mode that's easy to miss precisely because nothing actually fails: if a worker process dies, or the connection to the broker drops, a periodic task scheduled through Celery Beat simply never gets picked up. No exception is raised, because no code ever ran. No retry kicks in, because there was nothing to retry. The task is sitting in the queue exactly as scheduled, and from the outside, everything looks configured correctly. The only sign something's wrong is the absence of whatever that task was supposed to produce.

A heartbeat monitor catches this by watching for the thing that should have happened, not for an error that announces itself. The task pings a URL when it actually completes, and if that ping doesn't arrive on schedule, whether the cause was a crashed worker, a dead broker connection, or a bug in the task itself, you find out immediately. Here's how to set that up with the Downdar API.

What You Will Need

A Downdar API key from your account, a Celery app with a periodic task already scheduled through Celery Beat, and the requests library. Your API key should be read from the environment, never hardcoded into a task file.

Creating a Heartbeat Monitor

The /api/v1/heartbeats/create endpoint accepts a POST request with a name and returns the full heartbeat object, including the unique url your task will call on completion. This only needs to happen once. If scripting it isn't necessary for your workflow, creating the heartbeat from your Downdar dashboard and copying the URL works just as well.

# One-off setup, run from a shell or a Django management command. import requests   response = requests.post( 'https://downdar.com/api/v1/heartbeats/create', headers={'Authorization': 'Bearer YOUR_API_KEY'}, json={'name': 'Hourly inventory sync'} )   result = response.json()

The response looks like this:

{ "ok": true, "heartbeat": { "guid": "550e8400-e29b-41d4-a716-446655440000", "name": "Hourly inventory sync", "url": "https://downdar.com/heartbeats/550e8400-e29b-41d4-a716-446655440000", "paused": false }, "capacity": { "limit": 10, "used": 1, "remaining": 9 } }

Store the url as DOWNDAR_HEARTBEAT_URL in whatever config your Celery workers already read from.

Pinging the Heartbeat From Your Task

The ping goes at the end of the task, after the real work succeeds, and specifically outside of Celery's own retry handling, so a task that's mid-retry never reports a false success.

# tasks.py import os import requests from celery import shared_task   @shared_task(bind=True, autoretry_for=(ConnectionError,), max_retries=3) def sync_inventory(self): pull_latest_stock_levels() # raises normally; Celery's autoretry handles transient errors   # Only reached on a genuine successful completion, retries included. try: requests.get(os.environ['DOWNDAR_HEARTBEAT_URL'], timeout=10) except requests.RequestException: # Don't let a failed ping mark a successful sync as an error. pass

The autoretry_for and max_retries arguments mean transient errors, a dropped connection to the inventory API, for instance, get retried by Celery before the task gives up entirely. The heartbeat ping sits after that logic, so it only fires once pull_latest_stock_levels() has actually succeeded, whether that happened on the first attempt or the third. If every retry is exhausted and the task finally fails for good, the ping never fires, and the missing check-in is exactly the signal that should trigger an alert.

If Celery Beat is what's actually scheduling this task, double-check your beat schedule and the task name match exactly, a common way for tasks to silently stop running is a typo or a renamed task that beat never updates to reference correctly. A heartbeat monitor will catch that too, since a task that's never invoked will never ping.

Checking Heartbeat Status Programmatically

To surface a heartbeat's current state elsewhere, an internal dashboard or a Django admin page, the /api/v1/heartbeats/status endpoint returns just the essentials.

response = requests.post( 'https://downdar.com/api/v1/heartbeats/status', headers={'Authorization': f'Bearer {os.environ["DOWNDAR_API_KEY"]}'}, json={'guid': '550e8400-e29b-41d4-a716-446655440000'} )   status = response.json()['status'] # "unknown", "up", "warn", "down", or "paused"

That's enough to show "last inventory sync: healthy" somewhere your team already checks, rather than requiring a separate login just to confirm a background task is behaving.

A Note on API Key Security

Keep the API key out of your task files entirely and read it from the environment, the same way your Celery app already reads its broker URL.

# .env or your process manager's environment config DOWNDAR_API_KEY=your_api_key_here DOWNDAR_HEARTBEAT_URL=https://downdar.com/heartbeats/your_heartbeat_guid

The heartbeat URL is safe to sit alongside that config since it only accepts a check-in and reveals nothing about your account. The API key is what actually needs to stay out of task code, worker logs, and anything that isn't your own trusted environment.

Get Started

Start with whichever periodic task would cause the most damage if it silently stopped firing, a payment sync, a report generation, an inventory update. Once the ping is in place, a crashed worker or a broken beat schedule stops being invisible.

Start monitoring →