News for hackers. Stuff that matters.
Re:Web scraping via JavaScript runtime heap snapshots by BbzzbB on Friday April 29, 2022 @03:28PM (#31209642)
This is great, thanks a lot.

It's my understanding that Playwright is the "new Puppeteer" (even with core devs migrating). I presume this sort of technique would be feasible on Playwright too? Do you think there's any advantage or disadvantage of using one over the other for this use case, or it's basically the same (or I'm off base and they're not so interchangeable)?

I'm basing my personal "scraping toolbox" off Scrapy which I think has decent Playwright integration, hence the question if I try to reproduce this strategy in Playwright.

Re:Web scraping via JavaScript runtime heap snapshots by mdaniel on Friday April 29, 2022 @04:00PM (#31209972)
My understanding of Playwright is that it's trying to be the new Selenium, in that it's a programming language orchestrating the WebDriver protocol

That means that if you are running against Chromium, this will likely work, but unless Firefox has a similar heapdump function, it is unlikely to work[1]. And almost certainly not Safari, based on my experience. All of that is also qualified by whether Playwright exposes that behavior, or otherwise allows one to "get into the weeds" to invoke the function under the hood

1 = as an update, I checked and Firefox does have a memory snapshot feature, but the file it saved is some kind of binary encoded thing without any obvious strings in it

I didn't see any such thing in Safari

Re:Web scraping via JavaScript runtime heap snapshots by asabla on Friday April 29, 2022 @05:49PM (#31210949)
Well kind of for Firefox, there is this profiling tool which you could use (semi-built in)

https://github.com/firefox-devtools/profiler. Which let you save a report in json.gz format

Re:Web scraping via JavaScript runtime heap snapshots by nmstoker on Friday April 29, 2022 @07:19PM (#31211726)
I had understood that Playwright actually used the DevTools protocol rather than the WebDriver protocol, as mentioned here:

https://github.com/microsoft/playwright/issues/4862

And there's a bit of detail about how they're different here:

https://stackoverflow.com/q/50939116/142780

However that's more a detail and doesn't really undermine your point about Firefox / Safari being handled differently, it's just that Playwright implemented their own versions of the protocol for those two non-Chromium based browsers

Re:Web scraping via JavaScript runtime heap snapshots by nousermane on Sunday May 01, 2022 @07:25AM (#31224096)
> Firefox does have a memory snapshot feature, but the file it saved is some kind of binary encoded thing without any obvious strings in it

Those .fxsnapshot files are gzipped binary heaps. There is a 3rd-party decoder for it:

https://github.com/jimblandy/fxsnapshot