Good point, we should probably integrate that. Feel free to submit a PR!

BLAST can also be used to add automation to your own site/app FWIW.

> Feel free to submit a PR!

I think it's cool that you're experimenting in this area, but I'm not a huge fan of this as answer to a question about responsible/respectful web crawling. This stuff seems like it should be table stakes (even if you wanted to make it optional for the end user), but "yeah probably; learn the codebase, fork it, make changes, then we'll review it" really puts the onus onto the original poster.

Curious: if a user has an ad blocker, are they browsing responsibly?

Very. Malvertising is a thing. Adtech surveillance is a thing.

Ad blocker is the least user can do.

That's not why most users use ad blockers.

Ah you're right, my bad. Hope I didn't sound dismissive because I think some sort of robots.txt needs to exist for AI that's scraping the web both at train or test time.

I'm really not excited at all about the "scrape other people's data" use case for BLAST and if we can prevent it then awesome. I'm excited about BLAST automating science, legacy web apps, internal tools, adding AI automation to your own app, etc.

I will for my typical hourly rate plus the consulting bonus.

Feel free to take down your project until you comply with basic ethical standards.

The whole point of AI browser automation is mimicking human behaviour, fighting the anti-bot detection systems. If the point is interacting with systems, we'd be using APIs

...when there is an API. Often there isn't.

No, it's to automate tasks that can't be done using an API, like RPA.

you're saying essentiall the same thing as the parent comment, both are true

anti-bot fingerprinting exists to prevent you from using the human-facing site like an API

How can I find for myself an affordable accommodation in Iceland across several websites aggregating cheap accommodation but without signing API access contract with all of them and without building a middleware to abstract them?

I don't have API for _that_.

Seems Blast uses browser-use (https://github.com/browser-use/browser-use) which seems to be some client specifically for AIs to connect to/run browser runtimes.

Unfortunately, it seems like browser-use tries to hide that it's controlled by an AI, and uses a typical browser user-agent: https://github.com/browser-use/browser-use/blob/d8c4d03d9ea9...

I'm guessing because of the amount of flags, you could probably come up with a unique fingerprint for browser-use, based on available features, screen/canvas size and so on, that could be reused for blocking everyone using Blast/browser-use.

If calebhwin wanted to make Blast easier to identify, they could set a custom user-agent for browser-use that makes it clear it's Blast doing the browsing for the user.

Can browser-use be blocked using Anubis or other anti-bot measures?

I think these kind of requests can not be blocked. It is like asking if Claude computer control can be blocked from visiting websites. It is not detectable. You could ofc display captchas which are difficult to solve for an AI.

As mentioned, browser-use doesn't seem "out of this world" hard to fingerprint since all instances would be using the exact same settings.

Browser-use just switched to patchright https://github.com/Kaliiiiiiiiii-Vinyzu/patchright which is an “undetectable” version of Playwright and can rotate/randomize fingerprints. Another extension uses residential proxies and real fingerprints of real people.

Sorry if I'm missing something, but why can't Anubis not detect/block such AI agents? Is it because they use headful browsers?

Yes, they are doing the Proof of Work. If need be, we will completely emulate human-like mouse movements. These are essentially clients no different from real users.

But the thing is, the more they push for this, the less accessible websites will get, so there has to be another solution. Not everyone uses a mouse.

Source: I work in the industry.