Re:Applebot, the web crawler for Apple
by WorldWideWayne on Wednesday May 06, 2015 @11:52AM
(#9499550)
I wonder what are the legalities of this type of discrimination? Retail businesses aren't allowed to arbitrarily refuse service, they have to follow certain rules and be consistent.
[ Parent
| on HN ]
Is there legal backing to enforce obeying robots.txt, or is it a guideline?
Re:Applebot, the web crawler for Apple
by WorldWideWayne on Friday May 08, 2015 @10:56AM
(#9512029)
There are some court cases that involve robots.txt but I'm not sure because there was a lot of reading to do and I was hoping someone who knew would just come along and summarize it for us :)