Lightpanda: The headless browser designed for AI and automation

Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Friday January 24, 2025 @12:10PM (#42815171)

The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel.

Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by Tostino on Friday January 24, 2025 @12:38PM (#42815456)

You may reduce ram, but also performance. A good JIT costs ram.

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Friday January 24, 2025 @12:57PM (#42815601)

Yes, that's true. It's a balance to find between RAM and speed.

I was thinking more on use cases that require to disable JIT anyway (WASM, iOS integration, security).

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by Tostino on Friday January 24, 2025 @01:02PM (#42815636)

Yeah, could be nice to allow the user to select the type of ECMAScript engine that fits their use-case / performance requirements (balancing the resources available).

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by cxr on Friday January 24, 2025 @11:30PM (#42819510)

If your target is consistent enough (perhaps even stationary), then at some point "JIT" means wasting CPU cycles.

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by cush on Friday January 24, 2025 @03:14PM (#42816523)

> there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch)

> it's still ~10x better than Chrome

Do you expect it to stay that way once you've reached parity?

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Saturday January 25, 2025 @10:03AM (#42822006)

I don't expect it to change a lot. All the main components are there, it's mainly a question of coverage now.

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by nwienert on Friday January 24, 2025 @04:07PM (#42816943)

Playwright can run webkit very easily and it's dramatically less resource-intensive than Chrome.

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by fbouvier on Friday January 24, 2025 @06:16PM (#42817875)

Yes but WebKit is not a browser per se, it's a rendering engine.

It's less resource-intensive than Chrome, but here we are talking orders of magnitude between Lightpanda and Chrome. If you are ~10x faster while using ~10x less RAM you are using ~100x less resources.

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by bdhcuidbebe on Friday January 24, 2025 @08:13PM (#42818518)

How well does it compare to specialized headless scraper browsers, like camoufox (firefox based) or secret agent (chrome based)?

Either should reduce your ram usage compared to stock chrome by a lot.

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by whatevaa on Saturday January 25, 2025 @05:31AM (#42820785)

Careful, as you implement misssing features your RAM usage might grow too. Happened to many projects, lean at the beggining, get's just as slow when dealing with real world mesiness.

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by msoad on Saturday January 25, 2025 @07:19AM (#42821198)

Does it work nicely on Linux? I'm very curious about this

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by radium3d on Friday January 24, 2025 @08:52PM (#42818690)

As a web developer and server manager AI trainers scraping websites with no throttle is the problem. lol

[ Parent | on HN ]

Re:Lightpanda: The headless browser designed for AI and auto... by niutech on Wednesday January 29, 2025 @06:14AM (#42863707)

How about using QuickJS instead of full-blown V8? For example, Elinks has support for SpiderMonkey, QuickJS, MuJS: https://github.com/rkd77/elinks/blob/master/doc/ecmascript.t... and takes a few MB of RAM.

[ Parent | on HN ]