News for hackers. Stuff that matters.
Re:Web scraping case fails under Supreme Court's Dastar doct... by mc32 on Monday November 19, 2018 @05:14PM (#18490201)
There are a few forums where I don’t expect nor would prefer indexing. They may be niche forums, etc. Forums like reddit, of course I expect to be indexed. So, basically, it depends.
Re:Web scraping case fails under Supreme Court's Dastar doct... by jimktrains2 on Monday November 19, 2018 @05:38PM (#18490369)
Isn't this what robots.txt was created to convey?
Re:Web scraping case fails under Supreme Court's Dastar doct... by mc32 on Monday November 19, 2018 @06:05PM (#18490535)
Yes. But not all scrapers and indexers respect that? So it’s still a problem.
Re:Web scraping case fails under Supreme Court's Dastar doct... by baroffoos on Monday November 19, 2018 @08:44PM (#18491293)
Require a login/invite then