Re:Web Scraping in 2016
by Fiahil on Tuesday August 23, 2016 @05:00PM
(#12347599)
Sometimes, scraping a website is the only way at your disposal to fetch relevant information, you've paid for (or not). It could be simply the opening schedule of your local administration or the status of different pieces of public infrastructure.
If your administration don't have the resources (and it's often the case) to maintain a proper JSON API for you to fetch with a fancy python lib, then, it's not "super bad netizen stuff" to scrap a few HTML/PDF/XLS, parse them and display them for convenient public consumption on your personal website (and paying for the bandwidth).
It's 2016. State-companies holding a third party responsible for their own outages and poor planning is _bad faith_[1]. ETL? Never heard of it?
If your administration don't have the resources (and it's often the case) to maintain a proper JSON API for you to fetch with a fancy python lib, then, it's not "super bad netizen stuff" to scrap a few HTML/PDF/XLS, parse them and display them for convenient public consumption on your personal website (and paying for the bandwidth).
It's 2016. State-companies holding a third party responsible for their own outages and poor planning is _bad faith_[1]. ETL? Never heard of it?
[1]: https://citymapper.com/i/1208/soutenez-citymapper-et-lopen-d... (french)