News for hackers. Stuff that matters.
Re:What one may find in robots.txt by brazzy on Monday May 18, 2015 @08:01AM (#9563893)
How exactly is robots.txt a "restricted part"?
Re:What one may find in robots.txt by delinka on Monday May 18, 2015 @08:35AM (#9564033)
I believe the intent was that if robots.txt claims to disallow a section of a website, then French law would say that section is off-limits to visitors and visiting disallowed sections is punishable by law.

I doubt that this is a valid interpretation of the law. The robots.txt file simply mentions parts of the site that shouldn't be indexed, not which parts of a site shouldn't be viewed by the general public. You use a robots.txt file to prevent a search engine from following links that would enumerate all the possible dynamically-generated content on your site to conserve resources and to prevent junk results appearing for a search user.

Tangentially relevant: when you do have something you want indexed, it should probably be a static page that lives at a permanent URL. But never attempt to use a robots.txt file to "hide" sensitive data.

Re:What one may find in robots.txt by DanWaterworth on Monday May 18, 2015 @08:38AM (#9564040)
Off limits to robots, hence the name.

edit: I think you edited to clarify since I commented. Thanks.