News for hackers. Stuff that matters.
Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @01:24AM (#49930098)
I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

Re:Meta's Muse is fantastic for web scraping by bayindirh on Friday October 02, 2026 @01:35AM (#49930147)
Thanks for letting us know that we need a new layer of detection systems.

Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.

Humans are interesting creatures.

Edit: Please before assuming that I'm assuming things, this is an observation I'm making over time. It's possible that I'm in a bubble, but it's not a sample size of 1 (i.e. The comment I replied only).

Re:Meta's Muse is fantastic for web scraping by afro88 on Friday October 02, 2026 @01:59AM (#49930239)
I'm becoming more and more convinced that a big source of outrage on the internet is caused by people assuming that all other people are a homogenous blob.

It's not that "we" are going from one thing to another. It's that these are two different people, with different ethical boundaries.

Re:Meta's Muse is fantastic for web scraping by pcthrowaway on Friday October 02, 2026 @01:50AM (#49930196)
There is no detection method that will prevent AI from accessing systems without also blocking humans. The only thing we can do at this point is throttling.
Re:Meta's Muse is fantastic for web scraping by kees99 on Friday October 02, 2026 @02:06AM (#49930295)
Agreed on inevitable collateral blockage of real people using real "headed" browser. I'm getting a ton of that already, personally.

Throttling is poor help though. Mass scrapers are using "residential proxy" loophole + rotating UA and other attributes. You can't throttle somebody without identifying them. Unless you're talking about a global rate-limit.

Re:Meta's Muse is fantastic for web scraping by pcthrowaway on Friday October 02, 2026 @02:13AM (#49930334)
throttling based on sessions kind of works; we're headed in the direction that sites like Reddit will probably prevent logged-out users from viewing threads (as they already do with mobile devices)

Once the LLMs create sockpuppets to get around that, the web services will need to resort to profiling users more aggressively so that they know which actual human an account corresponds to.

If someone has a malicious browser extension that uses their session to scrape Reddit then, they're probably going to see significant usage obstacles.

We are headed to a very user-hostile place.

Re:Meta's Muse is fantastic for web scraping by mvt67 on Friday October 02, 2026 @04:21AM (#49931132)
Only if your life revolves around reddit.
Re:Meta's Muse is fantastic for web scraping by cyanydeez on Friday October 02, 2026 @09:14AM (#49933207)
I assume most people have see the photos of various "far east" people sitting at a bench with an array of 100 phones.

This predates AI as a _capitalism_ problem.

Re:Meta's Muse is fantastic for web scraping by Cakez0r on Friday October 02, 2026 @02:10AM (#49930314)
You're framing this as if people are deliberately making decisions that they believe are unethical. The reality is that people have different ethical frameworks. For example, I believe that there is no ethical distinction between whether a web request originates from a browser or from an LLM on my behalf.
Re:Meta's Muse is fantastic for web scraping by Kepten-Hook on Friday October 02, 2026 @03:27AM (#49930764)
Another person replied already and I agree with them, but also I think you need to see this from the perspective of the people operating the service you are accessing.

As an example, imagine a really small community maintaining a small site/wiki/cms/forum that has the ultimate goal of promoting human relationships around a common interest (let's say retro computing as an example, but it could be anything). There are many many many such communities on the internet.

It's very improbable you will specifically instruct your LLM to access their site, it's way more probable it will happen without you even knowing, as a result of you doing some /deep-research or something. And not only that, but your LLM will probably spawn a ton of agents to gather as much information as possible in as little time as possible. A torrent of requests will go at this community's site, effectively killing it. They are a small community, they use their spare time and money to maintain something to serve them, they don't have the resources to serve your LLM and until you showed up they probably never even had to think about Cloudflare. They are certainly not against you getting the content, but they don't want you causing them issues either and you just did.

End result? Your LLM (effectively you) DoSed a small community's site. You caused harm. Could you have caused similar harm if you were doing it on your own? Sure. But you would have done it on purpose, not accidentally while instructing your LLM to do something else.

This isn't a made up story, it has happened already more than 1 times.

So the question is, now that you know your LLM can cause harm without you even knowing it, how does this change your stance?

Re:Meta's Muse is fantastic for web scraping by mvt67 on Friday October 02, 2026 @04:24AM (#49931157)
Already have whatsapp and signal for this. Hardly need a website for these thing anymore.
Re:Meta's Muse is fantastic for web scraping by bayindirh on Friday October 02, 2026 @04:30AM (#49931186)
So shall we build more walls around our content and make it anti-FAIR?

We can stop putting information in the open in any form, as well.

FAIR: Findable, Accessible, Interoperable, Reusable.

Re:Meta's Muse is fantastic for web scraping by mvt67 on Friday October 02, 2026 @04:47AM (#49931289)
Why do you build a wall around your house genius?
Re:Meta's Muse is fantastic for web scraping by bayindirh on Friday October 02, 2026 @04:59AM (#49931356)
A public website is not a house, it's something between a big board in the city square or a community center where people can meet an interact.

Considering that, shall we paint the boards black or lock the doors to community spaces?

Also, while I assume this is not your first account here, these are worthy of reminding:

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

> Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.

For more, please refer to https://news.ycombinator.com/newsguidelines.html

Re:Meta's Muse is fantastic for web scraping by murderfs on Friday October 02, 2026 @05:39AM (#49931619)
The answer is that no one gives a shit: just look at people's views on adblock.
Re:Meta's Muse is fantastic for web scraping by dd8601fn on Friday October 02, 2026 @02:39AM (#49930484)
It’s a big question mark in the conversation.

Is defeating captchas unethical? I don’t think so. Not on its own.

Is scraping unethical? I don’t think so. Not on its own.

Are there tons of uses for both of those that are sketchy or outright wrong? Yeah, absolutely.

Re:Meta's Muse is fantastic for web scraping by watwut on Friday October 02, 2026 @03:55AM (#49930963)
> The reality is that people have different ethical frameworks.

Sure, Nazi, Hitler or Stalin all considered themselves to be the good guys. Kushner, Trump, etc are also acting within their ethical system (if I am getting money or power or fame it is ok to do it). Just about the only exception is Thiel who openly frames himself as evil, but is proud of it.

That does not mean we cant criticize their crappy actions or "ethical frameworks".

And yes, all the above are making deliberate decisions to be unethical assholes.

Re:Meta's Muse is fantastic for web scraping by cyanydeez on Friday October 02, 2026 @09:22AM (#49933264)
There's two types of cares:

1. People like other people.

2. Businesses need to sell product

The internet mixes those people, and an Agent basically pushes the signal to noise ratio that businesses have relied on the intenet to basically zero.

If the internet just allowed indescriminate traffic, neither #1 nor #2 survives, and while it's interesting to think the value is, it certainly isn't anything we understand.

Maybe you think the value will still exist, but some how agents will replace all value with equivelents. That's an argument, but I don't think it will.

Re:Meta's Muse is fantastic for web scraping by jareklupinski on Friday October 02, 2026 @08:45AM (#49932892)
humans also change over time

in batman, bruce wayne shuts does the massive surveillance network, seeing it was problematic (2008)

in spiderman, peter parker just shrugs off being able to know where anything is happening anytime (2026)

Re:Meta's Muse is fantastic for web scraping by bayindirh on Friday October 02, 2026 @08:50AM (#49932944)
> humans also change over time

Yes, that's what I'm referring to. Citing myself:

> Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.

This is exactly how I observe a shift in general.

Re:Meta's Muse is fantastic for web scraping by jareklupinski on Friday October 02, 2026 @08:59AM (#49933061)
maybe it'll go backwards? or the direction is chaotic / tick-tock
Re:Meta's Muse is fantastic for web scraping by bayindirh on Friday October 02, 2026 @09:12AM (#49933184)
I believe that the movement is sinusoidal,since every trend feeds its counterbalance.

OTOH, stabilizing at "0" or any point is impossible since the system is 2nd order and the response of the system is also an input to itself.

Re:Meta's Muse is fantastic for web scraping by jareklupinski on Friday October 02, 2026 @10:17AM (#49933812)
> sinusoidal

> OTOH, stabilizing

any function is stable; im just dissppointed if its predictable / manipulable / only has 2d

Re:Meta's Muse is fantastic for web scraping by Cakez0r on Friday October 02, 2026 @02:17AM (#49930354)
I think it's inevitable that eventually one of the big AI companies makes something like this. It's such an obvious consumer win. "I am not a bot" checkboxes are a string and peg tethering the elephant.
Re:Meta's Muse is fantastic for web scraping by onion2k on Friday October 02, 2026 @01:55AM (#49930218)
(using a custom side-loaded plugin that talks over websockets to a "driver")

Is that necessary? You could start Chromium with an open debug port and use Chrome Devtools Protocol to send commands.

Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:03AM (#49930277)
It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).
Re:Meta's Muse is fantastic for web scraping by raffraffraff on Friday October 02, 2026 @02:17AM (#49930353)
Does remote debugging port itself get detected, or does the presence of a webdriver connection to the port get detected? I believe it's the latter.

I've had success launching the browser and using dumb dumb methods to get around the captcha before attaching playwright.

Dumb dumb methods = wmctrl, xdotool, bash (work fine for Cloudflare's "are you human" check)

Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:24AM (#49930399)
I think it's the latter, and that's why I think using a "normal" non-debug browser is the best idea because it's essentially undetectable. And for the record, you can technically control Chromium using IPC if you feel like adding that feature & fully rebuilding it from scratch.
Re:Meta's Muse is fantastic for web scraping by LunaSea on Friday October 02, 2026 @02:07AM (#49930301)
Interesting, do you know what exactly changes on said root DOM node in case the remote debugging feature is enabled?
Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:10AM (#49930316)
`navigator.webdriver` afaik, but I think there's a couple more, I did some research on it a few months ago. If using Playwright/Puppeteer, they also have a few special things injected that you have to strip.
Re:Meta's Muse is fantastic for web scraping by Cakez0r on Friday October 02, 2026 @02:20AM (#49930376)
What model did you use to build this? I recently looked in to building something similar and got refusals.
Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:27AM (#49930415)
I built the POC by hand (just had like 3 things it could do: scroll, click, type), and then Codex used my established patterns to make it slightly more general and cleaned it up.
Re:Meta's Muse is fantastic for web scraping by LunaSea on Friday October 02, 2026 @04:24AM (#49931160)
Ah right, there are stealth plugins to remediate these types of hints I believe.
Re:Meta's Muse is fantastic for web scraping by pprotas on Friday October 02, 2026 @01:55AM (#49930216)
Camoufox bypasses most blocks with no problems https://camoufox.com/
Re:Meta's Muse is fantastic for web scraping by kurisufag on Friday October 02, 2026 @01:52AM (#49930201)
the 4get dev did the same thing a little while ago for his metasearch engine: https://git.lolcat.ca/lolcat/4play
Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:00AM (#49930248)
Yep, it's super similar to this! I think his is a bit overengineered, but tbh I haven't built Firefox plugins in forever so that might be the way you have to do stuff on FF. All you need is the websocket plugin to have "full access" to all visited webpages, and then your driver just hooks into the DOM like normal (with the extra ws layer—and even the websocket layer can be simplified away I think).
Re:Meta's Muse is fantastic for web scraping by d0vs on Friday October 02, 2026 @01:33AM (#49930138)
FYI Muse smashes through those captchas natively without prompting.
Re:Meta's Muse is fantastic for web scraping by kees99 on Friday October 02, 2026 @02:12AM (#49930330)
Some webshops (e.g. Aliexpress) already soft-block Chromium (at least Chromium-on-Linux). I.e. such users see larger-than-usual share of captchas.

Assuming "more people catching onto" this, expect Cloudflare and most everyone else to follow the suite.

Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:14AM (#49930337)
Are you saying they're blocking the `User-Agent`? Because that's trivial to bypass. I'm not exactly sure what you mean by "Chromium" because that's a whole class of browsers (Edge, Brave, Chrome, etc. are all forked "Chromium" browsers).
Re:Meta's Muse is fantastic for web scraping by kees99 on Friday October 02, 2026 @02:19AM (#49930368)
Aliexpress specifically uses a javascript blob to do detection, and quite a sophisticated one. There was a recent story about that blob messing with bluetooth headphones.

> not sure what you mean by "Chromium"

I meant that literally. This thing: https://www.chromium.org/getting-involved/download-chromium/...

Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @02:33AM (#49930440)
Here's my agentic Chromium browser browsing Aliexpress[1]. Usually I'd run it in headless mode, but wanted to test and make sure I'm getting the website proper.

It searched for "Aliexpress" on Google and clicked the link, hence the Google UTM params.

[1] https://imgur.com/a/eNulg11

Re:Meta's Muse is fantastic for web scraping by kotaKat on Friday October 02, 2026 @05:25AM (#49931541)
And the worst part is humans just getting more cognitively demanding captchas.

Some days I'm literally fatigued enough I can't clear the overly complex hCaptchas these days I keep getting forced upon me and bots sail right through this crap.

Re:Meta's Muse is fantastic for web scraping by frabcus on Friday October 02, 2026 @01:35AM (#49930143)
Right, but what happens when everyone uses that at scale? Without agreements and standards it isn't pretty.
Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @01:46AM (#49930182)
It's sad, but this train has left the station a quarter of a century ago imo. Many people have since become billionaires scraping the web without anyone agreeing (Google, Yahoo, and now possibly Anthropic and OpenAI).
Re:Meta's Muse is fantastic for web scraping by csnover on Friday October 02, 2026 @02:10AM (#49930315)
There is a huge difference between a clearly identifiable, robots.txt-respecting, largely symbiotic search engine spider versus this new AI-driven casual sociopathy of firing shotguns of deliberately masked bots at sites in ways which benefit no one except for, maybe, the bot-owner.

I agree with you, though, that the future has almost certainly already been decided, and see the eventual outcome of all this selfishness to be mandatory device attestation to access most services on the internet—which will of course never be allowed on any open platform. AI engineers who believed in freedom to compute (or just individual freedom more generally) have committed perhaps the greatest self-own in the history of our species to date.

Re:Meta's Muse is fantastic for web scraping by bel8 on Friday October 02, 2026 @01:40AM (#49930162)
Interesting, how is the chromium sandboxed? profile dir cli param? or deeper like chromium engine framework embeded in the application?
Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @01:42AM (#49930170)
Just simply a separate Chromium binary (not your usual browser).
Re:Meta's Muse is fantastic for web scraping by rjtc on Friday October 02, 2026 @01:49AM (#49930191)
anyone can spin these kind of side projects and do easy talk, but the moment you actually try to use this on signed-in Linkedin or Amazon its going to fail

the only solution is to drive your regular browser with all your sessions/cookies via an extension

Re:Meta's Muse is fantastic for web scraping by dvt on Friday October 02, 2026 @01:51AM (#49930198)
> the only solution is to drive your regular browser with all your sessions/cookies via an extension

This is exactly what I'm doing, but mocking/randomizing all the sessions/cookies/params (like resolution, OS, WebGL , etc.) in a separate Chromium binary. It's popular these days, but imo using your normal browser for agentic stuff is a very bad idea. These models do dumb stuff all the time.

Re:Meta's Muse is fantastic for web scraping by carsoon on Friday October 02, 2026 @01:55AM (#49930215)
no it doesn't fail. Agents are very good at comparing traffic characteristics from a real browser and a headless/automation browser and getting it to behave in the same manner. It's a cat and mouse game for sites stopping unauthorized access but right now llm agents are ahead.

For testing proxies should be used to avoid IP ban issues but given enough time modern agents can figure out how to bypass most of the modern scraping/automation prevention mechanisms.

I have built and used a lot of different automations and web scraping implementations for my business and it's never got permanently stuck yet, some take a bit longer, some shorter, but all within a reasonable time with little external help they have succeeded in their tasks.

Re:Meta's Muse is fantastic for web scraping by ares623 on Friday October 02, 2026 @01:30AM (#49930119)
By describing it here you've already released it no?
Re:Meta's Muse is fantastic for web scraping by busssard on Friday October 02, 2026 @04:24AM (#49931155)
please release it and call it SilverSurfer
Re:Meta's Muse is fantastic for web scraping by cyanydeez on Friday October 02, 2026 @09:10AM (#49933165)
I've got a plugin in development that'll do the same: via an extension, connect to a standard coding bot, so it can drive.

It can be used for testing software and using it as a human support.