Re:Colly: Fast and Elegant Scraping Framework for Golang
by asciimoo on Thursday October 05, 2017 @09:25AM
(#15408928)
Interesting idea, how do you imagine a channel based API for this?
[ Parent
| on HN ]
Re:Colly: Fast and Elegant Scraping Framework for Golang
by hellcow on Thursday October 05, 2017 @11:03AM
(#15409717)
I would ignore the GP's advice. Channels are prone to big errors -- panics and blocking -- which aren't detectable at compile time. They make sense to use internally but shouldn't be exposed in a public API. As one example, notice how the standard library's net/http package doesn't require you to use channels, but it uses them internally.
Re:Colly: Fast and Elegant Scraping Framework for Golang
by JoeAcchino on Thursday October 05, 2017 @10:15AM
(#15409332)
Would this work?
c := colly.NewCollector()
// this functions create a goroutine and returns a channel
ch := c.HTML("a")
e := <- ch
link := e.Attr("href")
// ...
I'm a bit rusty (ah!) with go, so bear with me if the above contains errors.
Re:Colly: Fast and Elegant Scraping Framework for Golang
by asciimoo on Thursday October 05, 2017 @10:34AM
(#15409488)
How do you recognize if the collector has finished? If the site doesn't contain "a" elements (e.g. because of a network error), this example would block forever.
Re:Colly: Fast and Elegant Scraping Framework for Golang
by dward on Thursday October 05, 2017 @10:59AM
(#15409677)
The producer closes the channel. This is differentiable from an open empty channel in a select.