Re:How to make fun of Google Bot (PicoLisp Wiki)
by Florin_Andrei on Friday July 15, 2011 @03:57PM
(#2768838)
Hold on, slash at the end is not standard?
[ Parent
| on HN ]
Re:How to make fun of Google Bot (PicoLisp Wiki)
by saalweachter on Friday July 15, 2011 @06:10PM
(#2769339)
No, I'm saying /21000/ will match a path with a directory named /21000 but not a file named /21000.
When I say "non-standard", I am saying am saying that if the website's URLs looked like "/21000/foo" and "/21000/foo?page=2", it would have been easier to craft a "Disallow" rule that would have successfully blocked all of the desired pages.