News for hackers. Stuff that matters.
Re:Auto-Scaling and Self-Defensive Services in Golang by aftbit on Sunday March 27, 2016 @03:07AM (#11369195)
When I needed to solve this problem, I spun up 4x as many workers as I needed and managed them through supervisor. This handles two of the requirements: 1. Autoscaling - Unless your workers are very expensive to run, or your load is exceptionally spiky (e.g. 100x jumps), just pre-scale instead of autoscaling. 2. If a process dies, supervisor (or runit or whatever) will restart it.

My supervisor config has each worker log to its own file (e.g. /var/log/acmeinc/worker_01.log). For death detection, I'd probably use a watchdog process that checks the mtime of each log file. If it's more than X minutes old, then no work has been done in that time, so kill the process and allow supervisor to restart it. Combine this with an in-process watchdog log message ("hey, I'm still alive!") on the main consumer thread every X minutes, and you have a fairly robust solution.

Tangentially, deferring engineering work by buying more hardware is almost always a good idea. :)

There are no comments on this comment yet.