Just in time. That XPath example is really useful for me.

I didn't finished the article yet, but I have one question. Is there ability in Scrapy to append new data after initial scrapping?

For example site at the moment of scrapping:

  Article 1  
  Article 2

Then we run scrapy again after some time and site will look like this:

  Article 1  
  Article 2  
  Article 3  
  Article 4

Can it just append article 3 and 4 to file/db table and stop scraping?

This project might give you some ideas https://github.com/TeamHG-Memex/scrapy-crawl-once.

That guy has quite a few scraping projects.

Scrapy is great. I have been using it at work to populate our solr index for our site search page. It is very extensible and powerful, but helps to understand Twisted if you need to do anything advanced.

Remember to scrape responsibly, check terms of service, robots.txt etc.

Why? Isn’t it better to use a vps or two (or even just a vpn), and strong rate limiting? I.e. try not to be noticed regardless of the robots.txt.

Because it's considered polite.

A lot of robots.txt exist to help crawlers automatically know which URLs are safe and which ones are mutable (i.e. the sign up link).

You should only ignore the robots.txt if you are very careful about what you are doing and have a very good reason for doing so.

I used Scrapy extensively last term in my non-CS co-op. Really loved how quickly I could make beautiful dashboards with a bit of Python / VBA.

My problem with Scrapy was that there was a lot of manual work involved for each site, but perhaps that speaks more about my proficiency rather than Scrapy.

What's the next step from here in terms of gathering, cleaning, and analyzing data?

Scrapy? Really?

Scrapie is a transmissible spongiform ecephalopathy, similar to mad cow disease, in sheep and goats.

https://en.wikipedia.org/wiki/Scrapie

The "Py" is for python.