Re:Meta's Muse is fantastic for web scraping
by dvt on Friday October 02, 2026 @02:03AM
(#49930277)
It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).
[ Parent
| on HN ]
Re:Meta's Muse is fantastic for web scraping
by raffraffraff on Friday October 02, 2026 @02:17AM
(#49930353)
Does remote debugging port itself get detected, or does the presence of a webdriver connection to the port get detected? I believe it's the latter.
I've had success launching the browser and using dumb dumb methods to get around the captcha before attaching playwright.
Dumb dumb methods = wmctrl, xdotool, bash (work fine for Cloudflare's "are you human" check)
I think it's the latter, and that's why I think using a "normal" non-debug browser is the best idea because it's essentially undetectable. And for the record, you can technically control Chromium using IPC if you feel like adding that feature & fully rebuilding it from scratch.
Re:Meta's Muse is fantastic for web scraping
by LunaSea on Friday October 02, 2026 @02:07AM
(#49930301)
Interesting, do you know what exactly changes on said root DOM node in case the remote debugging feature is enabled?
`navigator.webdriver` afaik, but I think there's a couple more, I did some research on it a few months ago. If using Playwright/Puppeteer, they also have a few special things injected that you have to strip.
Re:Meta's Muse is fantastic for web scraping
by Cakez0r on Friday October 02, 2026 @02:20AM
(#49930376)
What model did you use to build this? I recently looked in to building something similar and got refusals.
I built the POC by hand (just had like 3 things it could do: scroll, click, type), and then Codex used my established patterns to make it slightly more general and cleaned it up.