The State of Web Scraping in 2021

Re:The State of Web Scraping in 2021 by Quessked73 on Monday October 11, 2021 @04:45PM (#28832845)

Does anyone have a resource for getting into app-based scraping, if the API is obfuscated or rate limited?

[ Parent | on HN ]

Re:The State of Web Scraping in 2021 by nlh on Monday October 11, 2021 @08:11PM (#28834548)

ProxyMan on MacOS is quite awesome for this. It requires a bit of setup with certificates, etc., but once it's working you just fire up the target app on your phone and all the sweet sweet API requests appear on your big screen. I've scraped two apps very successfully this way.

It's also fascinating to see how developers-who-aren't-me setup their APIs when they assume that nobody's looking.

[ Parent | on HN ]

Re:The State of Web Scraping in 2021 by elorant on Monday October 11, 2021 @05:22PM (#28833251)

Run an http debugger, or some proxy, and find the endpoints.

[ Parent | on HN ]