News for hackers. Stuff that matters.

A comment on Build a crawler to crawl million pages with only one machine in just 2 hours. Parent comment

Re:Build a crawler to crawl million pages with only one mach... by staticautomatic on Tuesday February 28, 2017 @01:08PM (#13756008)
Celery will typically use lightweight "threads" like greenlets or eventlets. I don't think it use multiple processes insofar as we're talking about python where process == core.
[ Parent | on HN ]
Re:Build a crawler to crawl million pages with only one mach... by brianwawok on Wednesday March 01, 2017 @08:19AM (#13763428)
You can do greenlets or prefork concurrency. With prefork concurrency you get 1 process per fork. Gives you concurrency at the expense of a bit of memory.
[ Parent | on HN ]