I think this is exactly as petty and silly as last week's news, and now they get to spend a week explaining how this occurred and that they do obey robots.txt.

Bing failed the first part, Google the second. Both should fix that.

Why would anyone hosted on Windows have to specify every possible spelling variation to keep search engines out of a folder or file?

Here's another example:

http://www.ifma.org/robots.txt

These guys are disallowing /pv/

http://www.google.com/search?sourceid=chrome&ie=UTF-8...

You guys are indexing /PV/

/pv is /PV is /pV is /Pv

There is absolutely a reason. A user queried for a string, then followed a link. Biasing Bing's search results towards the followed links is a signal that improves their search.

> When students do this, it is called plagiarizing. The smoke getting thrown by MS is just to distract and divert while they scramble to hide what they did.

When I took exams, I wasn't allowed to consult with fellow students, read the internet, or open a textbook. My life as a developer would certainly be a lot different if I wasn't able to do any of those things now because it was "cheating" back when I was in school. It's an awful analogy and, as you've regurgitated it here, seems to be an effective smokescreen on Google's end.

Yes, but the actual effect is that they've just copied Google's results, rather than extracted valuable insight from their own users. And I don't see how Bing could have scraped the search terms and results from a Google session unless they are using code specifically tailored for Google's site. Given that, the clickstream excuse rings awfully hollow.

Plagiarism is going to be a judgement call in a developing field like search. Google's judgement is that Bing's scraping of their site, in this case, is fairly underhanded. I would agree.

They didn't prove anything. Every experiment needs a control. Where's Google's?

It's called an existential proof. If you want to prove that a black swan exists, all you need is 1 example of a black swan.

Strictly speaking, yes, non-controlled experiments exist. However without a control you cannot eliminate alternate explanations. Since we know that an alternate explanation exists here, the experiment doesn't show anything.

In the case of the black swan there's no credible explanation for that thing you see being anything but a black swan. The control is essentially Occam's razor.

If you want to explore a very different hypothesis, namely "Does Bing single out Google in its weightings of clickstream data?" then I suggest you go here:

http://projectgus.com/files/googlebing/seaport-trace.txt

That's a packet capture of some clickstream data. That should be more than enough to forge as much data as you like. You can then make up a nonsense search term, like "doesb1ngtrustgmorethananyoneelse" that should get zero results, then forge clickstreams going from a google.com search to "yes.com" as well as an equal number of searches for that term going from some other search engine to "no.com" and you can explore the weights as much as you want.

That said, I'd say that the fact that they weight Google highly enough that they'd take their word for it that a clearly irrelevant term should be mapped to some site is strong evidence, I think.

Mind you, I don't think it's "wrong" exactly for Bing to do this. I'm not worried about it destroying search, either. The spammers/SEO types will make it useless soon enough.

That said, I'd say that the fact that they weight Google highly enough that they'd take their word for it that a clearly irrelevant term should be mapped to some site is strong evidence, I think.

The evidence was that this occurred in 9% of the cases they tested. For terms that don't exist anywhere else on the net (ie the only place they could have been found anywhere in Bing's dataset was in the google query URL.)

Bing didn't weight Google ahead of a single other source of information, so I don't think this really comprises evidence of high weighting at all. To make that claim, someone needs to run a controlled test like the one you describe here.

spammers/SEO types will make it useless soon enough

Yeah, one thing that surprised me greatly is how easily it looks like this clickstream data could be injected. Although I don't really know what half the fields do, so there could be something clever going on in there.

Strictly speaking, I think you're right, but it was interesting that it would accept mappings that didn't exist elsewhere, rather than requiring the same mapping from at least one other source. But you're right that a more complete test is warranted. And Bing may have already adjusted things to prevent this, so we may never know.

> Although I don't really know what half the fields do, so there could be something clever going on in there.

True, but the mysterious parts I see appear to be constants (I may have overlooked something, though, so feel free to point out any mysterious dynamic bits I'm ignoring). Those should be out-and-out harvestable from actual Bing toolbars. With constants, I strongly suspect that we only need to harvest valid data, rather than figure out how to generate our own.

Worse for Bing, my experience in removing viruses says that "people who install lots of toolbars" and "people who get viruses/malware/joined to a botnet" are sets that overlap so a significant degree, so it might become hard to separate actual user clickstream data from forged clickstream data when they're both coming from the same, infected, computer.

Finally, I'm sorry I forgot to give you credit. If it's any consolation, I think I did give you credit last time I posted your link. Thank you for doing a better technical analysis than I've seen from anyone else on this story.

Ah, that's no problem. If you'd copied it to your domain and linked it there I might have minded, but if you're posting a link to my site I consider that plenty of credit.

I see. I would contend that that hypothesis is misleading at best, in bad faith at worst.