Furthermore, all bots worth their salt as far as threats go enable js and do everything they can to appear like a normal browser.
I'd love to know if puppeteer passed that test (probably). I have had exactly this problem many, many times and it is incredibly frustrating.
puppeteer passes the test if you run it from a machine that already has a good cloudflare reputation score. Try it from an AWS instance and it definitely fails 100% of the time.
(I've tried it, that's how I know)
There are github projects that are forks of things like selenium and puppeteer that are specifically designed to avoid detection for things like scraping google search results, etc.
Re:Blocked by Cloudflare
by tick_tock_tick on Wednesday August 09, 2023 @05:35AM
(#37060420)
That's fine as processing the javascript increases the coast at-least.
Blocking bots in the first place should not be acceptable since bots only act on behalf of humans. What should be blocked is abusive behavior that actually impacts the site - a single one off GET to what should be a static page should never be blocked, yet that's what CF does.
https://www.axios.com/cdn-cgi/challenge-platform/h/g/orchest...
It's amusing to see developers incessantly complain about writing source code that other people can read and understand and then we look at the web and see stuff like this which people are expected to run without question.
I'm not suggesting this type of JS payload is unique or that it's doing anything untoward. Nor am I suggesting with some effort one cannot determine what it's doing. I'm suggesting it does not follow this supposed ideal of writing code that others can easily read and understand. No one is expected to read it.
The heuristics used to attempt to differentiate between a so-called "bot" and a "human" are, IMHO, inadequate as long as there are "humans" that are mistaken for "bots" and blocked. "Use Chrome" is not a solution. A person using Firefox or some other non-Google software is still a "human". But not according to these brilliant "site protection" schemes. What level of false positives is acceptable.