Re:How to Crawl the Web Politely with Scrapy
by pmoriarty on Thursday August 25, 2016 @02:43PM
(#12361367)
Oh, you're talking about them inferring that you must have scraped them because you used or published data that only they had.
Not every scraper has publishing or using data in a detectable way as their motive.
For instance, I sometimes scrape a website to make an archive of it for my own personal use. I never publish the results or use them in any way that the website owners would ever know about. So the only way they could know that they were scraped is if I left some kind of scraping signature while scraping (such as scraping from a single IP and doing it quick enough to pop on their radar or perhaps regularly enough -- ie. without random waits between request, etc).
What you're talking about is probably mostly a concern to people/companies who are somehow making money from scraping data on other people's websites.
As for the game of cat and mouse, it lasts until they call in their lawyers. Then it's a game of "quit now or get destroyed".
But yes, if you can scrape the data without ever tipping off the company you're scraping, you can probably continue indefinitely, but you have to consider whether you can plausibly argue that you're getting that data from someplace else. If they sued you on the suspicion that you're scraping them, they'll probably subpoena the code to confirm that (or similar -- IANAL), and then proceed to try to make a case on things other than CFAA violations.