News for hackers. Stuff that matters.
Re:How to make fun of Google Bot (PicoLisp Wiki) by bauchidgw on Friday July 15, 2011 @08:42AM (#2767163)
see here for a spec how the robots.txt is parsed

http://code.google.com/web/controlcrawlindex/docs/robots_txt...

the robots.txt of http://picolisp.com (found at http://picolisp.com/robots.txt) allows the indexing of http://picolisp.com/21000 and all follow up pages.

why? see the spec:

The disallow directive specifies paths that must not be accessed by the designated crawlers. When no path is specified, the directive is ignored.

see here https://github.com/franzenzenhofer/robotstxt for coffeescript implementation of a robots.txt parser

Re:How to make fun of Google Bot (PicoLisp Wiki) by budgi3 on Friday July 15, 2011 @09:44AM (#2767318)
so what should his robots.txt look like? at the moment it is:

User-Agent: *

Disallow: /21000/

Re:How to make fun of Google Bot (PicoLisp Wiki) by saalweachter on Friday July 15, 2011 @10:46AM (#2767537)
It's mostly sufficient. /21000/ will not match "http://picolisp.com/21000, which is the first URL in the sequence, but the remaining URLs look like "http://picolisp.com/21000/!start?*Page=+2, so Googlebot will likely only continue to download a single page once it has re-read the robots.txt.

Which is what you deserve for using non-standard URL formats.

Re:How to make fun of Google Bot (PicoLisp Wiki) by Florin_Andrei on Friday July 15, 2011 @03:57PM (#2768838)
Hold on, slash at the end is not standard?
Re:How to make fun of Google Bot (PicoLisp Wiki) by saalweachter on Friday July 15, 2011 @06:10PM (#2769339)
No, I'm saying /21000/ will match a path with a directory named /21000 but not a file named /21000.

When I say "non-standard", I am saying am saying that if the website's URLs looked like "/21000/foo" and "/21000/foo?page=2", it would have been easier to craft a "Disallow" rule that would have successfully blocked all of the desired pages.

Re:How to make fun of Google Bot (PicoLisp Wiki) by bauchidgw on Friday July 15, 2011 @11:28AM (#2767715)

   User-Agent: *
   Disallow: /21000
or

   User-Agent: *
   Disallow: /