News for hackers. Stuff that matters.
Re:Show HN: (Sukhoi) Minimalist and Powerful Web Crawler in ... by gaojiuli on Thursday July 06, 2017 @10:44PM (#14715647)
What's the advantages of Sukhoi compared to the following.

Scrapy: https://github.com/scrapy/scrapy PySpider: https://github.com/binux/pyspider Gain: https://github.com/gaojiuli/gain

Re:Show HN: (Sukhoi) Minimalist and Powerful Web Crawler in ... by iogf on Friday July 07, 2017 @01:30PM (#14720259)
sukhoi is a little baby yet, it is built on a good concept in which it permits you to construct all kind of structures from your fetched data easily. It has a shorter learning curve due to its base concept too. The second example in sukhoi web page is extremely simple and it does something that if you are going to implement using other crawler it may get a bit tricky. Try conceiving the two examples in sukhoi using those web crawlers it should give you a feeling about how sukhoi makes things easier.
Re:Show HN: (Sukhoi) Minimalist and Powerful Web Crawler in ... by iogf on Thursday July 06, 2017 @02:41PM (#14712788)
No built-in session request implemented so far however, it can be easily emulated. Handling sessions is going to be implemented soon in websnake's framework.

https://github.com/iogf/websnake

Feel free to contact me on irc i hang at #vy on irc.freenode.org if you need help/willing to contribute somehow.

Re:Show HN: (Sukhoi) Minimalist and Powerful Web Crawler in ... by staticautomatic on Thursday July 06, 2017 @02:25PM (#14712638)
Can it be used with lxml's parser, and does it have support for what Requests calls `Session()`?
Re:Show HN: (Sukhoi) Minimalist and Powerful Web Crawler in ... by iogf on Thursday July 06, 2017 @02:42PM (#14712799)
and yeah, it could be used with lxml parser too as well as xpath(i'm planning to add that feature too).