pdufour

Link: https://github.com/pdufour/browser-use-wasm

So one of the big constraints of browser-use models is that they require a server running your vision language model to handle the images and convert it to actions.

That means if for instance you are a site owner and you want to include a AI widget that lets users control the webpage you are on via AI (i.e. ask the page to fill out this form) you would need a complicated server setup running a VLM.

I decided to build something different. We have had WebGPU and client-side models for a while, so I decided to build a library that does the following:

[Live page (iframe)] ──► [SnapDOM screenshot] ──► [ShowUI VLA WASM worker] ──► [DOM action at [x, y]]

Essentially this creates a browser-use model that runs entirely in your browser (no servers). There are a couple of libraries that make this possible:

- wllama for instance allows you to run any gguf model, which means easy access to VLA model on HF (I found ShowUi-2b to be the best but I want to try Nvidia LocateAnything)

- snapdom - as mentioned, this renders your webpage to an svg which is then passed to the VLA

After creating the workflow with those libraries, the rest is cake (not).

Some difficulties I had and my solutions for them:

- Snapdom had 1px rendering differences due to the inconsistencies rendering html that used a system font within a foreignObject tag in a svg - the fix it to use fonts from a CDN which provide font metrics for leading values

- Image resizing - you have to do some resizing to fit everything into limited space - this involved many different resizing methodologies

- Accuracy - finding out what increased my accuracy was quite hard at first till I found some evals such as MiniWoB++ (a web interaction test suite)

- Multi-step planning - my half-baked solution is to let the LLM generate the multiple steps, but in order for it to be comprehensive I would need to capture page, generate, capture page, generate, etc in a loop. I haven't done that yet

I am very interested in the client side LLM space so let me know if you have any thoughts!

"Just because you can doesn't mean you should." Wise advice I once got that I didn't follow. I like chasing problems that are quite difficult. Since 2024 I've wanted to be able to run a browser inside my browser. I know you could just do an iframe but I wanted to see if it was possible to render webpages inside webpages without that.

I knew Rust would be helpful here because of it's WASM compatibility. But it still required a lot of legwork to get it to work: DOM parsing, CSS layout, rendering, this all had to come together.

I started out using cheerpx which can run any x86 binaries, but couldn't find any browsers that would compile to that. So I abandoned that. I then moved onto using Servo (rust browser's engine) and its libraries like https://github.com/servo/stylo. https://github.com/DioxusLabs/blitz brought it all together and rendered pages via https://github.com/linebender/vello.

It's still not a true browser, it's missing a ton of features, but just rendering a webapge without using the DOM was fascinating to me.I learned coding it about Rust and WebGPU. Maybe it will be helpful for someone elses projects.

Check it out! https://github.com/pdufour/browserbrowserbrowser

Chaplin features this as well, via a "dispose" method. It'd be great to see a comparison list of all these Backbone meta frameworks.

Note: I am just a user of Chaplin, not an author.

This is nit-picky, but the huge vertical padding on the homepage is quite bothersome to me. I can fit just about one feature per page on my monitor. Something more condensed would look a lot better I think.

Same. Pleasantly surprised.

I worked / am working on something that Tyler discusses in his follow-up article.

It's called http://trackpile.com and it allows easy sharing from Spotify to Rdio, and vice versa.

Would love if you guys could check it out!

Try W3 Total Cache, they have super-tight Varnish integration.

> it's another to offer life advice based on what one perceives to be the norm.

But isn't that the case when anyone gives life advice? Everyone has their own own perceptions of norms.

Ah, that explains it. I was definitely noticing some similarities. Thanks for clearing that up.

Is this related to Lockitron?

Excellent article.

With Eric Reis' fame, I actually fear this may become standard practice at one point.

I hope this gets past a pilot episode.

Sweet, just registered.