News for hackers. Stuff that matters.
Re:So you want to write your own CSV code (2014) by franze on Friday May 15, 2020 @08:33AM (#23191392)
here the problem comes in, as people write

  User-agent: googlebot
  User-agent: bing    #behaved-badly
  User-agent: yandex
  Disallow: /
if # removes everythin afer the #, then it would be

  User-agent: googlebot
  User-agent: bing    User-agent: yandex
  Disallow: /
which results into the whole line " User-agent: bing User-agent: yandex" beeing thrown out as malformed, so only googlebot would be disallowed.

There are no comments on this comment yet.