News for hackers. Stuff that matters.

A comment on SeleniumBase: Python APIs for web automation and bypassing bot-detection.

Re:SeleniumBase: Python APIs for web automation and bypassin... by cruffle_duffle on Tuesday December 17, 2024 @11:01PM (#42448017)
As somebody who is now on the “need to scrape a website to get my customers data for them” side of the fence… I get the reason bot detection exists. If you want people to not scrape, offer API’s that allow customers or their software to log in using oauth and let their software / LLM agent grab their data for them.
[ Parent | on HN ]
Re:SeleniumBase: Python APIs for web automation and bypassin... by chii on Tuesday December 17, 2024 @11:21PM (#42448083)
> If you want people to not scrape, offer API’s

many sites want to prevent scrapers because they don't want their information aggregated - things like price lists and product availability etc.

I know groceries sites do this, to prevent customers from knowing price histories of products. They want to raise prices, then offer a discount to make it seem like the discount is legitimate.

[ Parent | on HN ]
Re:SeleniumBase: Python APIs for web automation and bypassin... by mintzworld on Wednesday December 18, 2024 @12:50AM (#42448402)
On the topic of scraping grocery sites, here's an example of bypassing bot-detection on Albertsons: https://github.com/seleniumbase/SeleniumBase/blob/master/exa... (A demo of that is in https://www.youtube.com/watch?v=Mr90iQmNsKM)
[ Parent | on HN ]
Re:SeleniumBase: Python APIs for web automation and bypassin... by bryanrasmussen on Wednesday December 18, 2024 @04:39AM (#42449194)
It seems weird to me that works - when I do scroll into views and similar behaviors in other code I do a random scroll speed to simulate human behavior, but SeleniumBase evidently doesn't.

Maybe I am just too paranoid.

[ Parent | on HN ]
Re:SeleniumBase: Python APIs for web automation and bypassin... by seleniumbase on Wednesday December 18, 2024 @07:37AM (#42449971)
SeleniumBase CDP Mode uses `DOM.scrollIntoViewIfNeeded` (https://chromedevtools.github.io/devtools-protocol/tot/DOM/#...), so it only scrolls when elements are offscreen, rather than always scrolling. This reduces the number of scrolls needed. Also, it seems that most anti-bot services are not looking at scrolling as a way of identifying users.
[ Parent | on HN ]