seleniumbase

In that video, I did some reverse engineering to create a stealth mode in an automation framework to bypass bot-detection and CAPTCHAs. It works from GitHub Actions Linux servers without needing a proxy (which is generally unheard of).

Looks like that one hasn't received any major updates in awhile.

The secret ingredient is to use Playwright's `connect_over_cdp()` method to connect to an existing browser that is already stealthy. Playwright can then perform its usual actions without being detected by anti-bot services. Example setup: `browser = playwright.chromium.connect_over_cdp("http://localhost:9222")`

Mix in some Python, PyAutoGUI, SeleniumBase, ThreadPoolExecutor, and then you get some serious multi-threaded CAPTCHA-bypass.

You can set `browser="firefox"`, but there's no stealth mode for it.

Here's an example that bypasses Kasada on the Hyatt website: https://github.com/seleniumbase/SeleniumBase/blob/master/exa...

It's OK. No offense was taken. It almost looked like the conversation was expanding into a "Python vs Java" debate, but (thankfully) it did not. I've seen both worlds. I've seen advantages to both. I decided to stay in the Python world.

Thank you!

The biggest issue with going from a home machine to a server is that you may lose having a "residential IP address", which is something that you'll want to have in order to prevent automation from being blocked outright. Hence the popularity of residential proxies. However, some servers live in a residential IP space, which makes them optimal for running web automation in. As was partially covered in https://www.youtube.com/watch?v=Mr90iQmNsKM, GitHub Actions appears to live in a "Residential IP space", which makes it a good server choice for web automation.

The "Python vs Java" debate is probably one for a different Hacker News post. :)

SeleniumBase CDP Mode uses `DOM.scrollIntoViewIfNeeded` (https://chromedevtools.github.io/devtools-protocol/tot/DOM/#...), so it only scrolls when elements are offscreen, rather than always scrolling. This reduces the number of scrolls needed. Also, it seems that most anti-bot services are not looking at scrolling as a way of identifying users.

Patching chromedriver is a lot easier than patching the browser. Plus, if you're just using a regular Chrome browser for the automation, then there's nothing to patch. Automated CDP calls aren't detectable if they don't leave any trace of automation activity. However, since Google created CDP, they might have ways of detecting automated CDP in ways that other services cannot.

There are live demos on YouTube of SeleniumBase bypassing CAPTCHAs (if you want see first): https://www.youtube.com/watch?v=Mr90iQmNsKM

There's actually a lot of examples being used for testing (https://github.com/seleniumbase/SeleniumBase/tree/master/exa...), which are run regularly (locally and in GitHub Actions). Plus, a lot of major companies are using SeleniumBase: https://github.com/seleniumbase/SeleniumBase/blob/master/hel... (if something breaks, I find out quickly)

There's a reCAPTCHA on the Pokemon website. This SeleniumBase example bypassed it: https://github.com/seleniumbase/SeleniumBase/blob/master/exa...

Call it "legacy code" if you'd like. That specific part is from a less common feature for setting options when running on a Selenium Grid. The new CDP Mode isn't compatible with The Grid (since CDP Mode makes direct CDP API calls without making Selenium API calls).

That patches chromedriver, (which gets renamed to uc_driver), but patching by itself isn't enough to bypass bot-detection. SeleniumBase also sets specific Chrome options and modifies methods to use the Chrome Devtools Protocol.

With SeleniumBase, you can bypass CAPTCHAs with one line of code: `sb.uc_gui_click_captcha()`

That method came from code that I accepted in a PR from December 31, 2019: https://github.com/seleniumbase/SeleniumBase/pull/459 Not a true representation of most of the code today.

SeleniumBase modifies the webdriver so that it doesn't get detected when used alongside the CDP stealth mode and methods. It'll download chromedriver for you. Not sure what you mean by the multiple branches, as there's just the primary one. What 1000-line methods are you referring to? By "flags", do you mean the different command-line options available? As for Playwright, they aren't undetected: See https://github.com/microsoft/playwright/issues/23884#issueco... - "Playwright is an end-to-end testing framework, where we expect you test on your own environments. Bypassing any form of bot protection is not something we can act on. Thanks for your understanding." On the contrary, SeleniumBase is OK with bypassing bot detection: https://github.com/seleniumbase/SeleniumBase/blob/master/exa...