Re:Ferret – Declarative web scraping
by thefounder on Tuesday October 02, 2018 @03:30PM
(#18124917)
This is so 2010...websites now use a mix of serverside and client side. You can't just watch http requests and figure out the "api" specifications. Just try to scarpe adwords/gmail/amazon etc. Consider that they may also use anti-scarping code on the client.
I just took a look at soundcloud and was able to get the data within a minute, as it's a basic setup. They use a json structure (most websites do) [1], and the data is fetched with a simply GET request.
If the website requires authentication, you just need to clone those requests too, and then you'll get some sort of cookie/session id back which you attach to any future requests.
As a bonus: You can then throw the json into a code converter online too which will convert your structure to a Type as well (useful if you are using a static language).
[1] https://imgur.com/a/Einoxdg