Re:How we reduced deployment times by 95%
by ahazred8ta on Wednesday August 28, 2019 @10:56PM
(#20826282)
> non-essential containers which exit don't get restarted or replaced
That's why you have one or two essential watchdog containers which relaunch the workers. You keep a large number of them in an "idle, but hot" status to allow for bursts?
Re:How we reduced deployment times by 95%
by bjacokes on Monday September 02, 2019 @01:56PM
(#20860739)
I'm a little confused by this approach; are there any non-essential containers in your suggested architecture? This sounds like the watchdog container is just a parent process that launches a bunch of subprocesses, which is definitely a workable solution, although not the one we decided to use. If there are primitives for an essential container to inspect container state and relaunch other containers in the task, that'd be great to know about.
The essential vs non-essential container designation is a little confusing. The standard use case for multi-container tasks seems to be that all containers are marked as essential, i.e. they essentially represent different services that are operating in concert on the same machine. This is definitely not the situation we're in, where each container is totally independent.
So it seems like we'd be a perfect use case for non-essential containers. However, (1) at least one container _must_ be marked as essential, and (2) non-essential containers which exit don't get restarted or replaced. This means we would still have a limited bleed-over effect (if the essential container exits, the other ones do too), and more importantly, we can't guarantee that our capacity will be robust to process exits.