News for hackers. Stuff that matters.
Re:Lightpanda: The headless browser designed for AI and auto... by JoelEinbinder on Friday January 24, 2025 @11:29AM (#42814724)
When I've talked to people running this kind of ai scraping/agent workflow, the costs of the AI parts dwarf that of the web browser parts. This causes computational cost of the browser to become irrelevant. I'm curious what situation you got yourself in where optimizing the browser results in meaningful savings. I'd also like to be in that place!

I think your ram usage benchmark is deceptive. I'd expect a minimal browser to have much lower peak memory usage than chrome on a minimal website. But it should even out or get worse as the websites get richer. The nature of web scraping is that the worst sites take up the vast majority of your cpu cycles. I don't think lowering the ram usage of the browser process will have much real world impact.

Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Friday January 24, 2025 @12:10PM (#42815171)
The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel.

Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

Re:Lightpanda: The headless browser designed for AI and auto... by Tostino on Friday January 24, 2025 @12:38PM (#42815456)
You may reduce ram, but also performance. A good JIT costs ram.
Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Friday January 24, 2025 @12:57PM (#42815601)
Yes, that's true. It's a balance to find between RAM and speed.

I was thinking more on use cases that require to disable JIT anyway (WASM, iOS integration, security).

Re:Lightpanda: The headless browser designed for AI and auto... by Tostino on Friday January 24, 2025 @01:02PM (#42815636)
Yeah, could be nice to allow the user to select the type of ECMAScript engine that fits their use-case / performance requirements (balancing the resources available).
Re:Lightpanda: The headless browser designed for AI and auto... by cxr on Friday January 24, 2025 @11:30PM (#42819510)
If your target is consistent enough (perhaps even stationary), then at some point "JIT" means wasting CPU cycles.
Re:Lightpanda: The headless browser designed for AI and auto... by cush on Friday January 24, 2025 @03:14PM (#42816523)
> there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch)

> it's still ~10x better than Chrome

Do you expect it to stay that way once you've reached parity?

Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Saturday January 25, 2025 @10:03AM (#42822006)
I don't expect it to change a lot. All the main components are there, it's mainly a question of coverage now.
Re:Lightpanda: The headless browser designed for AI and auto... by nwienert on Friday January 24, 2025 @04:07PM (#42816943)
Playwright can run webkit very easily and it's dramatically less resource-intensive than Chrome.
Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Friday January 24, 2025 @06:16PM (#42817875)
Yes but WebKit is not a browser per se, it's a rendering engine.

It's less resource-intensive than Chrome, but here we are talking orders of magnitude between Lightpanda and Chrome. If you are ~10x faster while using ~10x less RAM you are using ~100x less resources.

Re:Lightpanda: The headless browser designed for AI and auto... by bdhcuidbebe on Friday January 24, 2025 @08:13PM (#42818518)
How well does it compare to specialized headless scraper browsers, like camoufox (firefox based) or secret agent (chrome based)?

Either should reduce your ram usage compared to stock chrome by a lot.

Re:Lightpanda: The headless browser designed for AI and auto... by whatevaa on Saturday January 25, 2025 @05:31AM (#42820785)
Careful, as you implement misssing features your RAM usage might grow too. Happened to many projects, lean at the beggining, get's just as slow when dealing with real world mesiness.
Re:Lightpanda: The headless browser designed for AI and auto... by msoad on Saturday January 25, 2025 @07:19AM (#42821198)
Does it work nicely on Linux? I'm very curious about this
Re:Lightpanda: The headless browser designed for AI and auto... by radium3d on Friday January 24, 2025 @08:52PM (#42818690)
As a web developer and server manager AI trainers scraping websites with no throttle is the problem. lol
Re:Lightpanda: The headless browser designed for AI and auto... by niutech on Wednesday January 29, 2025 @06:14AM (#42863707)
How about using QuickJS instead of full-blown V8? For example, Elinks has support for SpiderMonkey, QuickJS, MuJS: https://github.com/rkd77/elinks/blob/master/doc/ecmascript.t... and takes a few MB of RAM.
Re:Lightpanda: The headless browser designed for AI and auto... by refulgentis on Friday January 24, 2025 @12:01PM (#42815086)
Generally, for consumer use cases, it's best to A) do it locally, preserving some of the original web contract B) run JS to get actual content C) post-process to reduce inference cost D) get latency as low as possible

Then, as the article points out, the Big Guns making the LLMs are a big use case for this because they get a 10x speedup and can begin contemplating running JS.

It sounds like the people you've talked to are in a messy middle: no incentive to improve efficiency of loading pages, simply because there's something else in the system that has a fixed cost to it.

I'm not sure why that would rule out improving anything else, it doesn't seem they should be stuck doing nothing other than flailing around for cheaper LLM inference.

> I think your ram usage benchmark is deceptive. I'd expect a minimal browser to have much lower peak memory usage than chrome on a minimal website.

I'm a bit lost, the ram usage benchmark says its ~10x less, and you feel its deceptive because you'd expect ram usage to be less? Steelmanning: 10% of Chrome's usage is still too high?

Re:Lightpanda: The headless browser designed for AI and auto... by JoelEinbinder on Friday January 24, 2025 @12:09PM (#42815163)
The benchmark shows lower ram usage on a very simple demo website. I expect that if the benchmark ran on a random set of real websites, ram usage would not be meaningfully lower than Chrome. Happy to be impressed and wrong if it remains lower.
Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Friday January 24, 2025 @12:15PM (#42815229)
I believe it will be still significantly lower as we skip the graphical rendering.

But to validate that we need to increase our Web APIs coverage.

Re:Lightpanda: The headless browser designed for AI and auto... by szundi on Saturday January 25, 2025 @06:42AM (#42821064)
Then came deepseek