News for hackers. Stuff that matters.
Re:Web scraping via JavaScript runtime heap snapshots by woodpanel on Friday April 29, 2022 @08:25PM (#31212154)
> is more ethical

How do you deal with pages that use JS to load their content (e.g. SERPs) and restrict those endpoints to be called from within that page?

I'm lucky if I can use cheerio to just traverse the DOM on a given page, but increasingly I have to render the page and that "scales" as well, at least in terms of maintainability since I can more or less use the same API to traverse the (then JS-modified) DOM

Re:Web scraping via JavaScript runtime heap snapshots by 10000truths on Friday April 29, 2022 @10:44PM (#31212902)
You can observe the network requests that the JS makes under the Network tab of the Developer Tools console. The restrictions you mention can be bypassed by setting the Origin and Referer HTTP headers to whatever satisfies the server.