It takes an interesting approach to crawling where
it renders the page using WebKit, and then exposes
the "rendered DOM" -- so that your crawling code
doesn't need to actually use JavaScript for
information extraction.
GoogleBot indexes content rendered by Javascript - even content delivered by an AJAX request. They've announced they are going to start penalizing sites that don't work well on mobile. I don't know the specifics of that (and they probably haven't shared them) but I do know that I've received automated email from Google Webmaster Tools and/or AdSense about one of my sites not working great on mobile: small UI elements grouped too closely together, content that's too wide, etc.
https://www.google.com/webmasters/tools/mobile-friendly/
According to them, starting april 21st it will be a ranking factor.
april 21st
Thanks for the link. I guess I better hop to it.
The distributed scrapy-cluster is the way to go, if you need to crawl anything of decent size ( maybe even Amazon - 300+ MM webpages, j/k :)
I see a lot of Python based projects recently, even in Bitcoin niche, we even have a local Toronto based Python meetup. Looks like Python dev community is active.
I have a domain name PYFORUM.com - would it be good idea to launch a forum site? With Bitcoin tipping built-in? So instead of saying "Thanks" people would be able to send $0.25 in Bitcoin to those who helped them in the forums or made them laugh? What are the most established Python forums out there?
Thanks!
https://github.com/scrapinghub/splash
People often use Scrapy + Splash together in the Python community for crawling more dynamic websites.
A team I collaborated with is also working on a project to make Scrapy usable in a "cluster context", it's called scrapy-cluster. The idea is scrapy workers running across machines and a single crawling queue (in the current prototype, powered by Redis) in between them all.
https://github.com/istresearch/scrapy-cluster