Web scraping via JavaScript runtime heap snapshots

Re:Web scraping via JavaScript runtime heap snapshots by 1vuio0pswjnm7 on Friday April 29, 2022 @06:35PM (#31211388)

Why doesn't the example chosen, YouTube, use something like Cloudflare "anti-bot" protection or Google reCAPTCHA.

When I request a video page, I can see the JSON in the page, without the need for examining a heap snapshot.

[ Parent | on HN ]

Re:Web scraping via JavaScript runtime heap snapshots by quickthrower2 on Friday April 29, 2022 @06:52PM (#31211526)

Because cloudflare, recaptcha etc. mean this is not general in possible. You need to quack like a normal user for it to work. If a site is really against scraping they could probably completely make it uneconomical by tracking user footprints and detect unexpected patterns of usage.

[ Parent | on HN ]

Re:Web scraping via JavaScript runtime heap snapshots by datalopers on Friday April 29, 2022 @06:55PM (#31211546)

They detect and block headless browsers just as easy.

[ Parent | on HN ]