Build a crawler to crawl million pages with only one machine in just 2 hours

Re:Build a crawler to crawl million pages with only one mach... by wopwopwop on Tuesday February 28, 2017 @12:51PM (#13755740)

I'm not an expert, but doesn't celery use multiple processes? I hope I'm not saying anything stupid, but I thought that's what "workers" were.

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by staticautomatic on Tuesday February 28, 2017 @01:08PM (#13756008)

Celery will typically use lightweight "threads" like greenlets or eventlets. I don't think it use multiple processes insofar as we're talking about python where process == core.

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by brianwawok on Wednesday March 01, 2017 @08:19AM (#13763428)

You can do greenlets or prefork concurrency. With prefork concurrency you get 1 process per fork. Gives you concurrency at the expense of a bit of memory.

[ Parent | on HN ]

Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @09:56PM (#13760575)

celery has a prefork pool that could take use of multiple processes.

[ Parent | on HN ]