News for hackers. Stuff that matters.
Re:So you want to write your own CSV code (2014) by franze on Friday May 15, 2020 @08:00AM (#23191130)
I coded a few robots.txt parsers i.e.: (these are just the ones i cared to publish) https://github.com/franzenzenhofer/simple-functional-robots-... https://github.com/franzenzenhofer/robotstxt

and even though robots.txt seems like a very, very simple text based protocoll there are unanswered mysteries

the biggest mystery is user agent groups and comments

i.e.:

  User-agent: googlebot
  User-agent: bing
  User-agent: yandex
  Disallow: /
so i am disallowing everything for google, bing, yandex; easy enough, but:

  User-agent: googlebot

  User-agent: yandex
  Disallow: / 
means that googlebot has no instructions, but yandex is disallowed all.

if a whole line is commented out

  User-agent: googlebot
  #User-agent: bing
  User-agent: yandex
  Disallow: / 
is a commented out line a blank line or a non existing line?

if it is interpreted as a blank linke, the disallow only counts for yandex, if the line is non existant, it counts for googlebot and yandex.

i like simple things, but sometimes the complexity is between the lines.

Re:So you want to write your own CSV code (2014) by iams on Friday May 15, 2020 @08:14AM (#23191218)

  User-agent: googlebot
  #User-agent: bing
  User-agent: yandex
  Disallow: /
why would it be interpreted as a blank line? If you remove everything after the #, that includes the new line characters at the end of the line. Leaving:

  User-agent: googlebot
  User-agent: yandex
  Disallow: /
Re:So you want to write your own CSV code (2014) by franze on Friday May 15, 2020 @08:33AM (#23191392)
here the problem comes in, as people write

  User-agent: googlebot
  User-agent: bing    #behaved-badly
  User-agent: yandex
  Disallow: /
if # removes everythin afer the #, then it would be

  User-agent: googlebot
  User-agent: bing    User-agent: yandex
  Disallow: /
which results into the whole line " User-agent: bing User-agent: yandex" beeing thrown out as malformed, so only googlebot would be disallowed.
Re:So you want to write your own CSV code (2014) by eyelidlessness on Friday May 15, 2020 @10:42AM (#23192729)
It's ambiguous, and this is why significant whitespace can be so frustrating. Unless it's specified, some people will interpret a full-line comment as a blank line with a comment ending it (`^#.$`), and others will interpret it as you have (`^#.?\n`). Neither is obviously correct (even if it's obvious to you).

Edit: I don't know how to escape in HN formatting. Obviously there are italics where literal asterisks should be.

Re:So you want to write your own CSV code (2014) by traes on Friday May 15, 2020 @07:14PM (#23198616)

  *** You can just use three asterisks. ***
* You can just use three asterisks. *

   Unfortunately you need something after them though. ***
Unfortunately you need something after them though.
Re:So you want to write your own CSV code (2014) by eyelidlessness on Saturday May 16, 2020 @10:26PM (#23208963)
Thanks, that's good to know!
Re:So you want to write your own CSV code (2014) by DoreenMichele on Friday May 15, 2020 @07:16PM (#23198626)
If you indent it two spaces, I think that should solve that problem:

   *(`^#.$`), and others will interpret it as you have (`^#.?\n`)*

(Not sure where you intended those asterisks. I made my best inference.)
Re:So you want to write your own CSV code (2014) by eyelidlessness on Saturday May 16, 2020 @10:27PM (#23208974)
Yeah I was hoping to keep them inline, but indenting surely would have helped. I intended them after the dots (`.*`, dot asterisk in case I get the escaping wrong, is 0 or more characters in regex).
Re:So you want to write your own CSV code (2014) by ThreeFx on Friday May 15, 2020 @08:10AM (#23191191)
What's the solution? Semantically it makes sense if it counts for both googlebot and yandex.

Have you observed one/both behaviours?

Re:So you want to write your own CSV code (2014) by franze on Friday May 15, 2020 @08:36AM (#23191421)
i am consulting a lot of aggregators, and we coose to interpret sometimes as this, sometimes in the other scenario. the way i interpret the spec, it is not defined behaviour.