News for hackers. Stuff that matters.

A comment on How to Crawl the Web with Scrapy. Parent comment

Re:How to Crawl the Web with Scrapy by estebarb on Monday September 13, 2021 @04:26PM (#28516420)
It is very slow. But personally, I prefer to write my crawlers in Go (custom code, not Colly).
[ Parent | on HN ]
Re:How to Crawl the Web with Scrapy by zatarc on Monday September 13, 2021 @04:32PM (#28516496)
Try Parsel: https://github.com/scrapy/parsel

It's way faster and has better support for CSS selectors.

[ Parent | on HN ]
Re:How to Crawl the Web with Scrapy by thegeekpirate on Tuesday September 14, 2021 @03:39AM (#28521530)
> But personally, I prefer to write my crawlers in Go (custom code, not Colly).

This is my current setup as well, been scraping on and off for 20+ years now.

[ Parent | on HN ]
Re:How to Crawl the Web with Scrapy by moehm on Monday September 13, 2021 @09:15PM (#28519008)
What's your problem with Colly? [0]

[0] http://go-colly.org/

[ Parent | on HN ]
Re:How to Crawl the Web with Scrapy by estebarb on Tuesday September 14, 2021 @03:18PM (#28529614)
Mostly that I started my crawler before learning about Colly and it didn't make sense to rewrite the code.

By "not Colly" I just wanted to remark that in Go is relatively easy to write a crawler from scratch.

[ Parent | on HN ]