A Web Crawler with Asyncio Coroutines

Re:A Web Crawler with Asyncio Coroutines by fabiandesimone on Tuesday September 15, 2015 @04:39PM (#10223025)

Thanks. Well, this does make me wonder if we are doing something wrong or we are performing actions that are slowing down the crawling. We have a good server (I believe).

[ Parent | on HN ]

Re:A Web Crawler with Asyncio Coroutines by troels on Tuesday September 15, 2015 @04:55PM (#10223147)

You might want to benchmark where your software is spending its time. A typical overhead is in the connection time. You may be able to speed things up with a local dns cache and by using http keep-alive. You also generally want to do a lot of parallel requests, since most time would be spent waiting for the subject site to respond.

[ Parent | on HN ]

Re:A Web Crawler with Asyncio Coroutines by kawera on Tuesday September 15, 2015 @04:47PM (#10223076)

Don't forget to check your connection too; maybe you are filling it up or have latency issues.

[ Parent | on HN ]

Re:A Web Crawler with Asyncio Coroutines by fsg7sdfg789 on Tuesday September 15, 2015 @06:25PM (#10223542)

I just reached out to you via email to see if I can help.

[ Parent | on HN ]