Re:80legs Web-Scale Apps Competition
by jdrock on Monday October 19, 2009 @06:14PM
(#891204)
I know Plura uses IP information to prevent bandwidth overages, but I don't think they have a list of top hosts.
80legs enforces throttles and rate limits on the sites that are crawled. We try to make a best guess at a rate limit for each site. Sometimes that best guess is too fast, so we take corrective action when we're notified by the webmaster. Users can't override the crawling rates.
80legs enforces throttles and rate limits on the sites that are crawled. We try to make a best guess at a rate limit for each site. Sometimes that best guess is too fast, so we take corrective action when we're notified by the webmaster. Users can't override the crawling rates.