News for hackers. Stuff that matters.
Re:Avoiding bot detection: How to scrape the web without get... by Aperocky on Sunday October 31, 2021 @08:21PM (#29061814)
Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.
Re:Avoiding bot detection: How to scrape the web without get... by Jugurtha on Sunday October 31, 2021 @08:42PM (#29061940)
>Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.

Can you say more about this? What do you mean by "Once you get to selenium it's usually over", and how do you manage cold starts in Selenium and emulating heavy usage?

Say your program starts right now, I assume you don't go through "adding heavy usage" to "warm-up", then get down to business, correct?

Re:Avoiding bot detection: How to scrape the web without get... by xnyan on Sunday October 31, 2021 @10:34PM (#29062511)
Selenium and other tools in that class essentially just build an api on top of a standard consumer browser engine(s). There are some differences that are difficult to completely hide, but it’s about as close to real as it gets and can be very difficult if not impossible to tell it’s an automation framework vs a standard web browser.

Travel information is also one of those services where it’s not weird for a significant number of their users to use it quite heavily, making behavioral detection more difficult.

Re:Avoiding bot detection: How to scrape the web without get... by arp242 on Sunday October 31, 2021 @10:52PM (#29062604)
By default Selenium exposes a few things in JS that are pretty trivial to detect, so you need to disable/hide that for starters. I don't know how easy or hard that is, but stock Selenium is a poor way to get around anti-bot stuff.
Re:Avoiding bot detection: How to scrape the web without get... by mhmmmmmm on Monday November 01, 2021 @04:16AM (#29064164)
Its fairly trivial, some stuff needs a modified browser executable or some JavaScript magic but it wont take you longer than 2-5 Hours to bypass most of the heuristics, disabling the window.navigator.webdriver flag and getting a residential IP is on its own enough to get single click captchas every 1-2 tries for example. To be fair I haven't looked into it since 2019 but i doubt that its gotten much harder.
Re:Avoiding bot detection: How to scrape the web without get... by northwest65 on Sunday October 31, 2021 @08:43PM (#29061952)
Yeah, but this was before Ajax when you had to hand roll XML RPC calls (i.e. long long ago).