News for hackers. Stuff that matters.
Re:Ask HN: What do people use to prevent crawlers? by pmx on Sunday July 23, 2017 @07:30AM (#14831768)
They generally are fairly slow but it doesn't matter because you just set it and leave it to it until it has all the data you want, leaving you to get on with other things.
Re:Ask HN: What do people use to prevent crawlers? by alexcnwy on Sunday July 23, 2017 @07:33AM (#14831781)
Plus very easy to parallelize
Re:Ask HN: What do people use to prevent crawlers? by amelius on Sunday July 23, 2017 @11:27AM (#14832672)
Unless the websites work together to protect themselves against crawlers that work that way (e.g., rejecting requests to different hosts from the same origin within a time interval).