News for hackers. Stuff that matters.
Re:Show HN: Crul || Query Any Webpage or API by wefarrell on Tuesday February 28, 2023 @03:47PM (#34974611)
In my experience bot detection is moving more towards looking at network activity and IP reputations. Using a proxy will go along way, it's easy to implement, and the cost can easily be passed onto the customer.
Re:Show HN: Crul || Query Any Webpage or API by pocket_cheese on Tuesday February 28, 2023 @04:23PM (#34975039)
In my experience, the thing that makes me actually have to lug out my ol' headless browser is that a lot of websites are starting to implement obfuscated cryptographic puzzles in their JS, making it really difficult to emulate without just running it in a browser.
Re:Show HN: Crul || Query Any Webpage or API by wefarrell on Tuesday February 28, 2023 @05:10PM (#34975634)
Starting to? That's been going on for at least the last 3-4 years. Akamai tends to rely on that more than Cloudflare and in my recent experience Akamai is winning that game and browser emulation alone, headless or not, isn't going to bypass Akamai. Recently I have seen browser emulation not be effective at all for bypassing bot detection.

The only kind of emulation where I have seen success is mobile and in that case you need to run a device emulator.

Re:Show HN: Crul || Query Any Webpage or API by asdadsdad on Tuesday February 28, 2023 @05:44PM (#34976000)
In my experience it's starting to moving towards "use a headless browser with patched attributes" or nothing else will work.

edit: I have quite a bit of experience with Akamai and other vendors =)