News for hackers. Stuff that matters.
Re:A Web Crawler with Asyncio Coroutines by reinhardt on Tuesday September 15, 2015 @04:25PM (#10222942)
It doesn't make much sense to give a number for speed without some specifics about the crawler environment, such as:

  - How many servers (if distributed)?
  - How many cores/server?
  - What kind of processing takes place for each page? 
    Does it just download and save the pages somewhere (local filesystem, cloud storage, database) or it extracts (semi) structured data? And so on.
Specifics aside, these days it's not hard to crawl millions of pages/day on commodity servers. Some related posts:

http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil...

http://blog.semantics3.com/how-we-built-our-almost-distribut...

http://engineering.bloomreach.com/crawling-billions-of-pages...

http://engineering.bloomreach.com/crawling-billions-of-pages...

Re:A Web Crawler with Asyncio Coroutines by fabiandesimone on Tuesday September 15, 2015 @05:10PM (#10223231)
Thank you very much!