Re:Blocked by Cloudflare
by sam0x17 on Wednesday August 09, 2023 @12:43AM
(#37058787)
Furthermore, all bots worth their salt as far as threats go enable js and do everything they can to appear like a normal browser.
[ Parent
| on HN ]
I'd love to know if puppeteer passed that test (probably). I have had exactly this problem many, many times and it is incredibly frustrating.
puppeteer passes the test if you run it from a machine that already has a good cloudflare reputation score. Try it from an AWS instance and it definitely fails 100% of the time.
(I've tried it, that's how I know)
There are github projects that are forks of things like selenium and puppeteer that are specifically designed to avoid detection for things like scraping google search results, etc.
Re:Blocked by Cloudflare
by tick_tock_tick on Wednesday August 09, 2023 @05:35AM
(#37060420)
That's fine as processing the javascript increases the coast at-least.