xitep

the page has been updated to point to the mirrored repo

The article is a very nice tutorial on asynch-io. I really enjoyed it. But as the authors themselves said the web-crawler they built in this article is just a toy.

Writing an efficient and _well behaved_ web crawler is imho quite a complicated undertaking. Others here have already pointed out that it's more or less a scalability problem, hence, a number doesn't make sense. reinhardt has provided a list of links - which from a quick glance - look very interesting and might bring you further.