Handling Worker Dyno Restarts

Dyno shutdowns, when scaling down, should be handled in the same manner as when they're shut down for daily application restarts, releases, rollbacks, etc. The approach depends on the nature of your application and your use case.

When Heroku attempts to shut down your dyno, it'll initially send it a SIGTERM. The idea is that you capture this signal to tell your worker dynos to stop accepting new jobs, finish the current job, and exit normally.

SIGKILL is sent 30 seconds after SIGTERM if the process hasn't terminated yet, in order to forcefully terminate it. If the job hasn't completed yet, you could either simply ignore it (and lose the job) or re-enqueue it before SIGKILL is sent so that a different worker can pick it up again.

When deciding to re-enqueue, you'll have to think about whether or not you mind having the entire job re-run (the simplest approach), or if the job needs to be implemented in an idempotent manner so that the next worker can effectively skip the work that has already been done and continue where the previous worker left off.

Some worker libraries might already handle SIGTERM in various ways, but you'll have to check if the one you're using does so and in what way.

The main thing to know is that you need to capture SIGTERM, which is issued when the dyno is about to shut down, after which the worker should stop accepting new jobs. The worker then has 30 seconds to either finish the job or re-enqueue it.