News for hackers. Stuff that matters.

A comment on SpiderKeeper: Admin UI for scrapy/open source scrapinghub.

Re:SpiderKeeper: Admin UI for scrapy/open source scrapinghub by dzink on Saturday March 07, 2020 @12:26PM (#22512745)
Are these spiders able to fetch content that loads in segments after scroll events?
[ Parent | on HN ]
Re:SpiderKeeper: Admin UI for scrapy/open source scrapinghub by jsmith99 on Saturday March 07, 2020 @02:58PM (#22513661)
Scrapy doesn't run javascript so it can't scrape client rendered content at all. There are projects that combine scrapy with a headless browser or javascript engine.
[ Parent | on HN ]