News for hackers. Stuff that matters.
Re:Cloudflare, Google, Bing Destroying the Infrastructure of... by MattGaiser on Wednesday August 26, 2020 @07:49PM (#24288626)
One possible problem with this is if one crawler gets a link to the website and then a parallel crawler gets a link to the website as well.

Enough parallel crawlers and you hit the captcha.

So the crawlers would need to be maintaining a record of sites and associated cookies. I don’t see why it couldn’t be done, but definitely adds to complexity.

There are no comments on this comment yet.