News for hackers. Stuff that matters.
Re:Build a crawler to crawl million pages with only one mach... by plantpark on Tuesday February 28, 2017 @10:04PM (#13760623)
Sorry for incomplete description in my article.

It's just a tutorial about how to use docker and celery to build a distributed system. I will have more test about the performance of multiprocess/threads/concurrencies or some other library that supports these technics. In this case, if you don't know how to or even don't want to build a test web server, some big sites like example.com could be your choice. BUT please be gentle to these public sites.

Thanks for your comment.

There are no comments on this comment yet.