Re:Colly: Fast and Elegant Scraping Framework for Golang
by stesch on Thursday October 05, 2017 @01:11PM
(#15410761)
Why are scrapers and scraping so popular? What is a real use case for it?
[ Parent
| on HN ]
Re:Colly: Fast and Elegant Scraping Framework for Golang
by troels on Thursday October 05, 2017 @03:45PM
(#15412039)
Most businesses I've worked for/with had one reason or another to scrape data from various places. Often it's watching competitors, but can also be part of an automation of some process that is already happening manually.
Re:Colly: Fast and Elegant Scraping Framework for Golang
by stephengillie on Thursday October 05, 2017 @01:35PM
(#15410933)
Not seeing ads.
Secondarily, there is a lot of data on the internet stored only in HTML pages. For data with multiple sources, HTML is still usually a common format. HTML just has more punctuation and errata to filter out than JSON, XML, or CSV.
Re:Colly: Fast and Elegant Scraping Framework for Golang
by stesch on Thursday October 05, 2017 @03:57PM
(#15412127)
So you want the data but you don't want to ask for it?
Re:Colly: Fast and Elegant Scraping Framework for Golang
by stephengillie on Thursday October 05, 2017 @04:11PM
(#15412226)
If I come across data I want/need, why should I have to highlight, copy, and read from my clipboard? It's much more elegant to have automation do this.
Re:Colly: Fast and Elegant Scraping Framework for Golang
by weberc2 on Thursday October 05, 2017 @04:51PM
(#15412490)
Accessing information that isn't available via an API. For example, much information on HN is only available via scraping. Many banks can't afford to rebuild their backend (probably largely due to the liability/compliance costs) to support an API/client model, but they still want an application--such applications are pretty much required to scrape said banks HTML.
Re:Colly: Fast and Elegant Scraping Framework for Golang
by conradfr on Thursday October 05, 2017 @05:08PM
(#15412602)
As an example:
I listen to the radio a lot and have always wanted to make a website with radio schedules like there is for TV. As this data is not available (at least not in France) I scrap [1] each radio website everyday to get it.
[1] Using https://github.com/rchipka/node-osmosis for now