Introduction to web scraping with Python

Re:Introduction to web scraping with Python by qrybam on Tuesday October 24, 2017 @06:58AM (#15540680)

I've been through the rigmarole of writing my own crawlers and and find Scrapy very powerful. I've run into roadblocks with dynamic/Javascript heavy sites; for those parts selenium+chromedriver works really well.

As parent and others have said: this is a grey area so make sure to read the terms of use and/or gain permission before scraping.

[ Parent | on HN ]

Re:Introduction to web scraping with Python by anewhnaccount2 on Tuesday October 24, 2017 @08:18AM (#15540990)

Notice how it's not a grey area when Google do it. The usually double standard apply I guess.

[ Parent | on HN ]

Re:Introduction to web scraping with Python by davvolun on Wednesday October 25, 2017 @12:02PM (#15551251)

I don't understand what this comment is referring to. Google's spider respects robots.txt, just block all paths and google will not crawl your site. So too for Bing, Yahoo, Baidu (some complications though, I think), Yandex.... Most of the major spiders respect robots.txt.

Is there some major Google web scraping effort I'm not aware of?

[ Parent | on HN ]

Re:Introduction to web scraping with Python by wootie512 on Tuesday October 24, 2017 @08:50AM (#15541149)

I am just getting into web scraping and have also been using Selenium with either Firefox or PhantomJS. Is there a better way to handle the javascript heavy sites? I found one library called dryscraping but haven't had the time to look too deep into it.

[ Parent | on HN ]

Re:Introduction to web scraping with Python by RhodesianHunter on Tuesday October 24, 2017 @09:02AM (#15541200)

Take a look at the Google team's Puppeteer

https://github.com/GoogleChrome/puppeteer

[ Parent | on HN ]

Re:Introduction to web scraping with Python by brianwawok on Tuesday October 24, 2017 @09:30AM (#15541356)

Splash runs in docker and does a decent job. From the scrapinghub team.

[ Parent | on HN ]