Re:Build a crawler to crawl million pages with only one mach...
by aaroninsf on Tuesday February 28, 2017 @01:22PM
(#13756197)
I use Celery inside Docker, mostly for the lazy-ops advantages; makes it very simple to bring up new pools of Celery workers, shut them all down, and mix projects on the same host to maximize its utilization.
Generally I'm getting fond of containers as a mechanism to encapsulate deployments e.g. in Python which have a lot requirements and which I've found finicky to make portable.
Full disclosure: I do something even worse, have containers which pull updates when I like with deploy keys, and run Celery etc in a virtualenv in the container... :P
The latter feels truly shameful but it does make it easy to keep the project contained even when running outside a container...
Generally I'm getting fond of containers as a mechanism to encapsulate deployments e.g. in Python which have a lot requirements and which I've found finicky to make portable.
Full disclosure: I do something even worse, have containers which pull updates when I like with deploy keys, and run Celery etc in a virtualenv in the container... :P
The latter feels truly shameful but it does make it easy to keep the project contained even when running outside a container...