News for hackers. Stuff that matters.

A comment on Scrapy: New Python web crawling & scraping framework (built on Twisted). Parent comment

Re:Scrapy: New Python web crawling & scraping framework (bui... by tdavis on Sunday December 28, 2008 @11:30AM (#412006)
wget is synchronous while Twisted is an asynchronous networking engine. This means that you don't need to wait for a request to finish before making another one (or making pancakes, or doing whatever you want).

I essentially wrote a parallelized version of scrapy which has the ability to make hundreds of requests per second, depending on available CPUs. You could never achieve that level of performance using wget.

[ Parent | on HN ]
Re:Scrapy: New Python web crawling & scraping framework (bui... by breck on Sunday December 28, 2008 @07:37PM (#412380)
This is great. I was running threads on a current crawl job but the real bottleneck is BeautifulSoup and not the network. So splitting the project into threads(while it helped about 10%) wasn't really necessary and Twisted probably would have done the trick.
[ Parent | on HN ]