News for hackers. Stuff that matters.
Re:Ask HN: What are best tools for web scraping? by kanishkalinux on Tuesday November 14, 2017 @08:00AM (#15694593)
for mostly static pages requests/pycurl + beautifulsoup more than sufficient. For advance scraping, take a look at scrapy.

for javascript heavy pages most people rely on selenium webdriver. However you can also try hlspy (https://github.com/kanishka-linux/hlspy), which is a little utility I made a while ago for dealing with javascript heavy pages for simple usage.

There are no comments on this comment yet.