News for hackers. Stuff that matters.
Re:Robots.txt meant for search engines don’t work well for w... by tomjen3 on Friday April 21, 2017 @06:49AM (#14164422)
They also make the content available to the public, which is directly competing with the site owners.

Thats inviting lawsuits they can't win and expecting people to pay the bandwidth for it too.

There are no comments on this comment yet.