Re:So you want to write your own CSV code (2014)
by iams on Friday May 15, 2020 @08:14AM
(#23191218)
User-agent: googlebot
#User-agent: bing
User-agent: yandex
Disallow: /
why would it be interpreted as a blank line? If you remove everything after the #, that includes the new line characters at the end of the line. Leaving: User-agent: googlebot
User-agent: yandex
Disallow: /
Re:So you want to write your own CSV code (2014)
by franze on Friday May 15, 2020 @08:33AM
(#23191392)
here the problem comes in, as people write
User-agent: googlebot
User-agent: bing #behaved-badly
User-agent: yandex
Disallow: /
if # removes everythin afer the #, then it would be User-agent: googlebot
User-agent: bing User-agent: yandex
Disallow: /
which results into the whole line " User-agent: bing User-agent: yandex" beeing thrown out as malformed, so only googlebot would be disallowed.
Re:So you want to write your own CSV code (2014)
by eyelidlessness on Friday May 15, 2020 @10:42AM
(#23192729)
It's ambiguous, and this is why significant whitespace can be so frustrating. Unless it's specified, some people will interpret a full-line comment as a blank line with a comment ending it (`^#.$`), and others will interpret it as you have (`^#.?\n`). Neither is obviously correct (even if it's obvious to you).
Edit: I don't know how to escape in HN formatting. Obviously there are italics where literal asterisks should be.
Re:So you want to write your own CSV code (2014)
by traes on Friday May 15, 2020 @07:14PM
(#23198616)
*** You can just use three asterisks. ***
* You can just use three asterisks. * Unfortunately you need something after them though. ***
Unfortunately you need something after them though.
Re:So you want to write your own CSV code (2014)
by eyelidlessness on Saturday May 16, 2020 @10:26PM
(#23208963)
Thanks, that's good to know!
Re:So you want to write your own CSV code (2014)
by DoreenMichele on Friday May 15, 2020 @07:16PM
(#23198626)
If you indent it two spaces, I think that should solve that problem:
*(`^#.$`), and others will interpret it as you have (`^#.?\n`)*
(Not sure where you intended those asterisks. I made my best inference.)
Re:So you want to write your own CSV code (2014)
by eyelidlessness on Saturday May 16, 2020 @10:27PM
(#23208974)
Yeah I was hoping to keep them inline, but indenting surely would have helped. I intended them after the dots (`.*`, dot asterisk in case I get the escaping wrong, is 0 or more characters in regex).
and even though robots.txt seems like a very, very simple text based protocoll there are unanswered mysteries
the biggest mystery is user agent groups and comments
i.e.:
so i am disallowing everything for google, bing, yandex; easy enough, but: means that googlebot has no instructions, but yandex is disallowed all.if a whole line is commented out
is a commented out line a blank line or a non existing line?if it is interpreted as a blank linke, the disallow only counts for yandex, if the line is non existant, it counts for googlebot and yandex.
i like simple things, but sometimes the complexity is between the lines.