News for hackers. Stuff that matters.
Re:Meta's Muse is fantastic for web scraping by onion2k on Friday October 02, 2026 @01:55AM (#49930218)
(using a custom side-loaded plugin that talks over websockets to a "driver")

Is that necessary? You could start Chromium with an open debug port and use Chrome Devtools Protocol to send commands.

Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:03AM (#49930277)
It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).
Re:Meta's Muse is fantastic for web scraping by raffraffraff on Friday October 02, 2026 @02:17AM (#49930353)
Does remote debugging port itself get detected, or does the presence of a webdriver connection to the port get detected? I believe it's the latter.

I've had success launching the browser and using dumb dumb methods to get around the captcha before attaching playwright.

Dumb dumb methods = wmctrl, xdotool, bash (work fine for Cloudflare's "are you human" check)

Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:24AM (#49930399)
I think it's the latter, and that's why I think using a "normal" non-debug browser is the best idea because it's essentially undetectable. And for the record, you can technically control Chromium using IPC if you feel like adding that feature & fully rebuilding it from scratch.
Re:Meta's Muse is fantastic for web scraping by LunaSea on Friday October 02, 2026 @02:07AM (#49930301)
Interesting, do you know what exactly changes on said root DOM node in case the remote debugging feature is enabled?
Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:10AM (#49930316)
`navigator.webdriver` afaik, but I think there's a couple more, I did some research on it a few months ago. If using Playwright/Puppeteer, they also have a few special things injected that you have to strip.
Re:Meta's Muse is fantastic for web scraping by Cakez0r on Friday October 02, 2026 @02:20AM (#49930376)
What model did you use to build this? I recently looked in to building something similar and got refusals.
Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:27AM (#49930415)
I built the POC by hand (just had like 3 things it could do: scroll, click, type), and then Codex used my established patterns to make it slightly more general and cleaned it up.
Re:Meta's Muse is fantastic for web scraping by LunaSea on Friday October 02, 2026 @04:24AM (#49931160)
Ah right, there are stealth plugins to remediate these types of hints I believe.