Re:Cloudflare, Google, Bing Destroying the Infrastructure of...
by MattGaiser on Wednesday August 26, 2020 @07:49PM
(#24288626)
One possible problem with this is if one crawler gets a link to the website and then a parallel crawler gets a link to the website as well.
Enough parallel crawlers and you hit the captcha.
So the crawlers would need to be maintaining a record of sites and associated cookies. I don’t see why it couldn’t be done, but definitely adds to complexity.
Enough parallel crawlers and you hit the captcha.
So the crawlers would need to be maintaining a record of sites and associated cookies. I don’t see why it couldn’t be done, but definitely adds to complexity.