Gracefully shutdown worker dynos with sigterm

Dyno shutdowns, when scaling down, should be handled in the same manner as when they're shutdown for daily application restarts, releases, rollbacks, etc.

The approach depends on the nature of your application and your use-case.

When Heroku attempts to shutdown your dyno, it'll initially send it a SIGTERM. The idea is that you capture this signal in order to tell your worker dynos to stop accepting new jobs, finish the current job, and exit normally.

SIGKILL is sent 30 seconds after SIGTERM if the process hasn't been terminated yet, in order to forcefully terminate it. If the job hasn't completed yet you could either simply ignore it (and lose the job) or re-enqueue it before SIGKILL is sent so that a different worker can pick it up again.

When deciding to re-enqueue you'll have to think about whether or not you mind having the entire job re-run (the simplest approach), or if the job needs to be implemented in an idempotent manner so that the next worker can effectively skip the work that has already been done, and continue where the previous worker left off. This could be significantly more complicated to implement depending on what you're doing.

Some worker libraries might already handle SIGTERM in various ways, but you'll have to check if the one that you're using does so, and in what way.

The main thing to know is that you need to capture SIGTERM, which is issued when the dyno is about to shutdown, after which the worker should stop accepting new jobs. The worker then has 30 seconds to either finish the job, or to re-enqueue it.