News for hackers. Stuff that matters.
Re:Spidermon: Scrapinghub’s Now Open Sourced Spider Monitori... by bryanrasmussen on Tuesday March 05, 2019 @12:45PM (#19312409)
How does this handle sites that are rendered entirely or nearly entirely by JavaScript client side? My only real concern when looking at a new scraping solution, almost everything else can be worked with.
Re:Spidermon: Scrapinghub’s Now Open Sourced Spider Monitori... by landyman on Tuesday March 05, 2019 @03:31PM (#19313945)
I haven't used Spidermon yet, but Scrapinghub has a Scrapy plugin called Splash which can be used to render JS pages within a Scrapy spider.