Re:A Web Crawler with Asyncio Coroutines
by greglindahl on Wednesday September 16, 2015 @02:51AM
(#10225043)
We used djbdns on every crawl machine, and did not find DNS to be limiting at all. You should also make sure there isn't any connection tracking, or firewalls/middleboxes which are doing connection-based anything, or NAT, or really anything other than raw Internet between you and the Internet.
[ Parent
| on HN ]
There are no comments on this comment yet.