News for hackers. Stuff that matters.
Re:Web scraping via JavaScript runtime heap snapshots by marwis on Friday April 29, 2022 @04:33PM (#31210265)
This sadly does not help if js code is minified/obfuscated and data is exchanged using some binary/binary-like protocol like grpc. Unfortunately this is increasingly common.

The only long term way is to parse visible text.

Re:Web scraping via JavaScript runtime heap snapshots by mdaniel on Friday April 29, 2022 @06:05PM (#31211109)
I've never seen grpc from a browser on a consumer-facing site; do you have an example I could see?

That said, for this approach something like grpc would be a benefit since AIUI grpc is designed to be versioned so one could identify structural changes in the payload fairly quickly versus the json-y way of "I dunno, are there suddenly new fields?"

Re:Web scraping via JavaScript runtime heap snapshots by marwis on Friday April 29, 2022 @06:48PM (#31211490)
Not aware of any actual grpc websites but given grpc-web has 6.5k stars on github something must be out there.

Google's websites frequently use binary-like formats where json is just an array of values with no properties, and most of these values are numbers. See for example Gmail.

Re:Web scraping via JavaScript runtime heap snapshots by kccqzy on Friday April 29, 2022 @11:26PM (#31213091)
That's likely just an encoding into JSON of protobuf messages. The typical wire format contains a sequence of field tag followed by contents. They can just make it an even-sized array.