How to Monitor a Rails/Sidekiq Job

Steven Carver
Blog
Tutorials

Sidekiq's default retry behavior is genuinely useful and genuinely dangerous at the same time. A job that starts failing gets retried automatically, with increasing delays, for up to 25 attempts spread across roughly three weeks, before it finally lands in the dead set. That's three weeks where a scheduled job, say, a nightly billing reconciliation, can be silently failing and retrying in the background while everything looks fine from the outside, unless someone happens to be watching the dead queue in the Sidekiq Web UI, which most teams check reactively, not proactively.

A heartbeat monitor sidesteps that entirely: the job pings a URL only when it actually completes, and if that ping doesn't show up on schedule, you're alerted immediately, not three weeks and 25 retries later. Here's how to wire that up for a Sidekiq job using the Downdar API.

What You Will Need

A Downdar API key from your account, and a Rails app with Sidekiq already configured, typically scheduled through sidekiq-cron, sidekiq-scheduler, or a plain cron entry that enqueues the job. Your API key should be stored as a Rails credential or environment variable, never committed to your repository.

Creating a Heartbeat Monitor

The /api/v1/heartbeats/create endpoint accepts a POST request with a name and returns the full heartbeat object, including the unique url your job will call on completion. You only need to do this once per job. If you'd rather not script it, creating the heartbeat from your Downdar dashboard and copying the URL works just as well, the API is mainly worth using if you're provisioning monitoring as part of a deploy process.

# One-off setup, run from a Rails console or a rake task. require 'net/http' require 'json'   uri = URI('https://downdar.com/api/v1/heartbeats/create') request = Net::HTTP::Post.new(uri) request['Authorization'] = 'Bearer YOUR_API_KEY' request['Content-Type'] = 'application/json' request.body = { name: 'Nightly billing reconciliation' }.to_json   response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) { |http| http.request(request) } result = JSON.parse(response.body)

The response looks like this:

{ "ok": true, "heartbeat": { "guid": "550e8400-e29b-41d4-a716-446655440000", "name": "Nightly billing reconciliation", "url": "https://downdar.com/heartbeats/550e8400-e29b-41d4-a716-446655440000", "paused": false }, "capacity": { "limit": 10, "used": 1, "remaining": 9 } }

Store the url as an environment variable, something like DOWNDAR_HEARTBEAT_URL.

Pinging the Heartbeat From Your Job

The ping belongs at the very end of perform, after the actual work is done, so it only fires on a genuine successful completion, not on an attempt that's about to be retried.

# app/workers/billing_reconciliation_worker.rb require 'net/http'   class BillingReconciliationWorker include Sidekiq::Job   def perform reconcile_billing_records # raises normally if something goes wrong   # Only reached if the line above didn't raise or get retried. ping_heartbeat end   private   def ping_heartbeat uri = URI(ENV.fetch('DOWNDAR_HEARTBEAT_URL')) Net::HTTP.get_response(uri) rescue StandardError => e # A failed ping shouldn't fail the job itself, just log it. Rails.logger.warn("Heartbeat ping failed: #{e.message}") end end

This ordering is what makes the monitoring reliable rather than misleading. Because reconcile_billing_records runs first and is allowed to raise, Sidekiq's normal retry mechanism handles failures exactly as it always would, and the heartbeat is never touched during a failed attempt. Only a job that fully completes, on whichever attempt that turns out to be, reaches the ping. And the ping itself is wrapped in its own rescue block, so a network hiccup reaching Downdar can't cause a successful job to log as an error in Sidekiq.

Checking Heartbeat Status Programmatically

To surface a heartbeat's current state elsewhere in your app, say, on an internal ops dashboard, the /api/v1/heartbeats/status endpoint returns just the essentials.

uri = URI('https://downdar.com/api/v1/heartbeats/status') request = Net::HTTP::Post.new(uri) request['Authorization'] = "Bearer #{Rails.application.credentials.downdar_api_key}" request['Content-Type'] = 'application/json' request.body = { guid: '550e8400-e29b-41d4-a716-446655440000' }.to_json   response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) { |http| http.request(request) } status = JSON.parse(response.body)['status'] # "unknown", "up", "warn", "down", or "paused"

That's enough to show "last reconciliation: healthy" wherever your team already looks, instead of everyone needing a separate login to check.

A Note on API Key Security

Rails credentials are the cleanest place for the API key on a typical Rails app, since they're encrypted and already excluded from version control by default.

# Add via: EDITOR=vim rails credentials:edit downdar_api_key: your_api_key_here   # Usage anywhere in the app Rails.application.credentials.downdar_api_key

The heartbeat URL itself doesn't need the same treatment, it only accepts a check-in and reveals nothing about your account, so a plain environment variable is fine for that one. The API key is the part worth keeping out of logs, error trackers, and anything that isn't your own server.

Get Started

Pick the Sidekiq job that would cause the most damage if it quietly stopped running, and start there. Once the ping is in place, Sidekiq's own retry logic keeps doing its job, and you find out the moment it finally runs out of chances, not weeks later.

Start monitoring →