Re:How My Popular Site was Banned by Google
by robryan on Tuesday October 11, 2011 @07:05PM
(#3100806)
Would be easy for the google bot to work this out to, replace the URL part that looks like a dynamic variable with different values and see if it returns a page for them all like a search engine would. Surprised it doesn't appear to do this.
Re:How My Popular Site was Banned by Google
by StrawberryFrog on Tuesday October 11, 2011 @05:42PM
(#3100429)
> Actually there is no difference between search page result sites like the OP and what you said
Uh, yes there is. http://www.somesite.com/index.html?q=foo is obviously a search (right down to the choice if q for query), but http://www.somesite.com/buy_foo_now.html does not look like one.
You might say that technically they could have the same back-end, but the web is flexible - just about any URL scheme could resolve to the same back-end. The difference is that the first is honest and the second one goes out of its way to hide what it does.
Re:How My Popular Site was Banned by Google
by ceejayoz on Tuesday October 11, 2011 @07:03PM
(#3100798)
> Uh, yes there is. http://www.somesite.com/index.html?q=foo is obviously a search
Obviously? Drupal's default non-rewritten URLs are index.php?q=foo, where foo is the page path. `q` can and does stand for more than one thing.
Actually there is no difference between search page result sites like the OP and what you said. Having a URL that ends in .html does not mean there is a static html file on the server. For examples, nextag and pricegrabber have URLs like blah.com/digital/Canon-EOS-7D-SLR-Digital-Body/m739295014.html. Scroll around these sites and you'll see they're anything but static.
Each page is simply a result of a query. Whether the database is local (like nextag) or remote (like Amazon API query like OP) is inconsequential. Personally, I would be VERY happy if Google hid/ignored these kind of search-query sites. I have not once found any of these sites useful. Problem is the large gray area in which these sites operate and what I find useless, someone else might find useful.