Re:The (humorous) dangers of Google's new deep search
by neilc on Thursday April 17, 2008 @06:43PM
(#166762)
It's not "their way of doing things", it is just the way the web works (per HTTP spec). Any crawler would have done the same thing in that situation -- that fact that it happened to be Google is merely coincidental. Given the scale at which they operate, you can't expect Google or any other web-scale crawler to be mind-readers.
What happens when we're dealing with interfaces that are a little bit more ill-defined? Will Google continue to demand that you follow their way of doing things?
Google's attitude in this case suggests they will.