News for hackers. Stuff that matters.

A comment on Nike.com's robots.txt. Parent comment

Re:Nike.com's robots.txt by JoeyBananas on Friday February 11, 2022 @10:43AM (#30302696)
A parser that can't handle comments doesn't care about obeying the robots.txt in the first place.
[ Parent | on HN ]
Re:Nike.com's robots.txt by marginalia_nu on Friday February 11, 2022 @11:17AM (#30303165)
There is an RFC, but in practice the format is super informally specified. You'll encounter

# comments

// comments

<!-- comments -->

/* comments */

plain text lines as comments

TeX-markup

script tags

HTML

PHP code

Whatever this is: https://www.costablancaworld.com/robots.txt

search-engine specific directives (Yandex has a few strange ones)

[ Parent | on HN ]
Re:Nike.com's robots.txt by supercoffee on Friday February 11, 2022 @12:09PM (#30303911)
> https://www.costablancaworld.com/robots.txt

Somebody screwed up and made their robots.txt in a rich-text editor like wordpad.

[ Parent | on HN ]
Re:Nike.com's robots.txt by colejohnson66 on Friday February 11, 2022 @12:06PM (#30303854)
> Whatever this is: https://www.costablancaworld.com/robots.txt

That's an RTF file that was saved as a TXT (without removing formatting)...

https://en.wikipedia.org/wiki/Rich_Text_Format

[ Parent | on HN ]