Re:RoboBrowser: Your friendly neighborhood web scraper
by pkmishra on Thursday May 12, 2016 @02:50AM
(#11681815)
What benefit does it provide in comparison to Scrapy?
[ Parent
| on HN ]
Re:RoboBrowser: Your friendly neighborhood web scraper
by alexroan on Thursday May 12, 2016 @04:11AM
(#11682034)
From what I can tell only recently starting to uzse Scrapy is that alot more "magic", shall we say, happens in the background so long procedures which could be a few hundred lines using bs4/requests/mechanize/etc can be minimized into a lot less. Looking at Robobrowser, it seems like it will reduce some of the coding effort but not to the extent that Scrapy does.
Re:RoboBrowser: Your friendly neighborhood web scraper
by kmike84 on Thursday May 12, 2016 @05:38AM
(#11682319)
I think the main difference is that Scrapy is async - it downloads pages in parallel by default, so it is more efficient. But async APIs can be harder to use - you need callbacks or generators everywhere, so sync packages (like RoboBrowser) can be easier to get started.