I would not equate scraping with periodic searching for a keyword.

The problematic part is when you scrape data off of websites, and the owners don't want you to do it; as in, they would not even be happy if you manually copied that stuff into an excel file for fun or profit.

I can see that, although automating to decrease views of a page(from checking by refresh)... I suppose the end result is sign up in this case.

So for me I'll set it up for them and them run it.

> How do you deal with the legality of scraping?

How did Google deal with it, when they started their search engine business?

My counter to this is, without Google's "algorithm" the search results would not be great/find what you want. And in this case websites want to volunteer/agree to index their site for visibility but then you still have to visit the site because conveniently The site summary ends before the content that you want. I don't know how I would find websites unless referred on some website.

I kinda felt the same way about how Netflix started out but I think they probably had a deal to pay a portion of revenue from dvd's that they rented out.

robots.txt