Once you tell the world that FOO and BAR are related, and the whole world looks at it and says "Yay, it IS related!", the relationship stops being ephemeral and becomes actual, reflected in actions of the users. You no loner own that relationship.
You can't claim ownership of facts, even if you discovered them first, or created them into existence.
It's not as straightforward as all that. Suppose you write a book. The existence of that book is a fact. The words that are in that book are now a fact. Now, I create a book that says, "The following book was written by DennisM, this is a list of the words that are in that book, in order" and then precede to duplicate your book. Those are the words that are in the book, it's a fact, no one can deny that it's a fact. That's not defensible, even though I'm just stating a fact.
We're in a somewhat grey area here, dealing with ethical issues that have never been clearly dealt with before. However, I feel that a general principle of the web applies: If I create some data, and don't at least implicitly give you permission to use it, it's unethical for you to use it.
This is the central idea behind copyright, behind robots.txt, behind plagiarism, and even behind privacy.
I wrote a book, in which I described some new way to dance. Before the book was published, the ideas were mine to keep secret. After the book was published, a bunch of readers picked up on the idea. They started all dancing in certain way, and someone described their behavior. That description will be de-facto copy of my book, and yet it is entirely legitimate. Because users own they behavior, not the author of the book who inspired them.
Now if someone simply copied pages from my book, that would be a copyright violation. But that's not what happened. As it is, it's an original work of art. Would that suck for me as a dance-inventor? Obviously. Do I want to live in a society where description of my behavior is owned by the person who inspired or directed it? Absolutely not.
Your idea of data ownership is contrary to tradition and contrary to the law. Facts (such as relationship between FOO and BAR) can not be owned in any way shape or form, except as a trade secret. You can not copyright a fact, or trademark a fact. In some cases you can patent application of a fact to a problem, but that's not the case here, as you can't and don't want to patent relationship between a word and target page.
Just because you put effort into something, does not mean it's yours. You probably wish it were true, but again, that's not how the law works, and not what the tradition is. You can't own facts.
As to "the central idea of copyright", bear in mind that in addition to not being able to copyright ideas or data, there's also the notion of "fair use".
Explain this to me. In terms of raw bytes it's probably more data than every book that has ever been written. It was certainly more expensive to write than any particular book. What is the important difference here?
To make it a little more concrete, suppose someone was doing the same thing with google's streetview data. Now, it's an empirical fact that if you go to such-and-such an address, and look around, it looks like the pictures that google took. Does that mean it's ok to use those pictures, just because their resemblance to reality is factual?
You're referring to the aggregate, so far as I know copyright law works on specific instances. Let's say the Kindle had a button that allowed you to isolate a couple of sentences and broadcast them to your friends on Facebook, or wherever. The small portion of the text being shared would fall under fair usage and not be a copyright infringement violation, even though the aggregate of all Kindle users might be large.
What is the important difference here?
I think the biggest difference between the two sides debating this issue is that one side thinks the amount of effort Google put into creating search results is a relevant concern, while my side thinks that once those results are shared with users Google's legal/moral claims to any ownership of that data evaporate.
The streetview point highlights the problem. If Google and I both take pictures of the Church of Mxyzptlk at 14th and Main, we both own copyright to our respective pictures. The fact that I created my picture after "piggybacking" off of a Google search for Mxyzptlk is irrelevant to copyright. So far as know (IANAL) the combination of a link with search terms is a utilitarian set of data rather than a creative expression and hence not copyrightable.
There's a more general point here: If something is difficult to do but easy to copy, then society should prevent that copying so that the creator can be rewarded for their effort. This maximises creativity and productivity and boosts GDP. Musicians, inventors, artists, authors, drug companies and software developers all rely on this principle. If we didn't have copyright and patents and robots.txt then there would be no incentive to produce half of the things people want, and we would all be much worse off. Google (and any other search engine) should be subject to the same rules.
It would be great for Google and Microsoft to both come clean about all the different factors they use in their search but I'm not holding my breath.
The comparison to a web directory is perfect - in the absence of a robots.txt Google and other crawlers think the data is absolutely fair game. There isn't yet a corresponding analogy for clickstream data so its fair game. Are you sure all of Google's tools respect robots.txt for passive analysis of user initiated actions on other websites?
>You're addressing the letter of the "law" not the spirit. Do we really need a new formal standard to indicate that this is unethical? To me it's pretty plain from the ethical standards the web has already agreed upon.
But this whole "no idea how they got there" is kind of bogus. The whole notion of link analysis is to infer from a link that a page is important. Now they're infering from a click that page is important. The relative importance of that page is a function of the search query, like the relative importance of a link is a function of the page from which it originated.
Link analysis and click throughs are both second order effects, right? If CNN.com starts linking to a page, you don't really know why that page is important, except for the fact that CNN.com now points to it.
So lets be clear MS knows how it got there as much as link analysis does. Its because the second order effect of a click implies relevance, just as a link does.
There are two pieces of information in a click, one provided by google, and the other provided by the user.
Google says, "'foo.com' is a good result for [foo]."
The user says, "yup."
Which is providing more information here?
Everyone at google is happy to admit that if the internet didn't exist, we would have a very hard time ranking it. Is bing ready to admit that if google didn't exist, they'd have a hard time ranking the internet?
The user provides at least two pieces of information.
1) The search query (which you attmpted to hide in Google's information)
2) A URL they retrieve from Google
Google provides one piece:
1) An ordered list
By FAR the most important piece of information is the search query.
In fact, given a choice between having the search query or an arbitrary ordered list, I'm sure Bing would value the search query the most.
Clickstream data from Google is almost certainly but one piece of clickstream data (even if you are special-cased). Just as Microsoft.com is just one page you index. Now you wouldn't say that you'd have time ranking the internet if Microsoft.com didn't exist would you? Of course not.
And I think this is where Google is going astray. It's fine to call a spade a spade, but your overreaching. Trying to argue that MS couldn't index w/o Google is simply absurd. And if Matt was trying to be diplomatic, you're certainly not being by implying that MS would have a hard time indexing w/o Google.
That's not an accurate view of how search works. The user provides the query, but the results that are returned don't need to have anything obvious to do with the query. The terms might not be on the page, and might not be found in any data remotely associated with the page. We don't just intersect posting lists here.
Furthermore, with suggest and instant, often the query itself is provided by google.
Fair enough point about suggest and instant. In those cases Google is adding value to the query, but I think we'd both agree that it is incrementally so. Instant generally just helps you get to the eventual query faster. Spell checking does add value, but its a +epsilon. In either case, your right, we should add this to our function.
In any case I think my point still holds. The query is the most important aspect. The browsed page, for indexing purposes is the second most important aspect. The association between the query and the URL is the least important aspect.
IOW, the least value for Bing is the search algorithm. The query and page to add to the index (as a browsed page) are the most important.
I don't understand why you believe this. This is the entire basis for why search is hard. It is the hard problem that search tries to solve. Many companies have spent in aggregate, billions of dollars in R&D trying to solve this, and all but a few have folded. It's an "AI-complete" problem in that solving it perfectly would be a sufficient demonstration of strong AI. It's the whole reason why Google exists.
Now the reason I say this is that to my best approximation the key difference that makes Google better than Bing (when it is) is the index size, and not the search algorithm.
So the two key pieces of value are: A) queries that people don't do on your site, which would have generated poor relevance
B) Pages that are actually browsed that you don't have indexed or up to date.
Now don't get me wrong. There is value in the association, but I think I'd capture 80% of the value with the above. Now as you point out, these lists are often non-obvious, but Bing does a near equal job of creating the lists. And if you give them the search terms where they have gaps. And fill the index, then I think the gap closes most of the way.
And lets be clear, because they clicked the link doesn't mean its a good link. See ehow.com or expertsexchange. But it does have some value.
And frankly I think MS would have been willing to let it go if Google had gone straight to MS with it and said, "We have this data. Even though its not unethical we can spin it to the media to make it look bad. Kill the association." MS probably would have as the net win isn't that huge. Now the PR loss in pulling it would be worse than the PR loss in keeping it (with no integrity loss, since they [and I] think it is perfectly ethical).
I'm pretty thoroughly sure this is false. I'm sorry I can't show data to back it up, but Google has many many systems that are years ahead of bing's technology. I'm kinda hamstrung here by being unable to reveal anything about them. To a decent approximation, both Google and Bing likely have the whole internet that matters (and that they are allowed to crawl) in their indexes. Bing is just unable to return this data for as many queries as Google is.
>but Bing does a near equal job of creating the lists.
To the extent that this is true, how much of it is due to data harvested from Google search? This is something that google can't really demonstrate with evidence, and what I would have hoped that bing would clarify in a public statement if they had anything defensible to say.
Maybe this is true, but its not apparent. I had commented on this a month or so back that I thought Bing was better at "vague" queries, where I don't know exactly what I'm looking for. But I'll know it when I see it.
Whereas Google is really good at targeted queries. I need info on the HP battery model number 003D434F90. These searches in Bing will often bring back literally nothing, while Google will often have one or two links, but they happen to be the link that I want. The text of the query is almost always found in these pages.
From that I infer it is index size, since the text is in the page.
To the extent that this is true, how much of it is due to data harvested from Google search?
While I find the quality of searches similar I don't find the results to be similar, if that makes sense. If Bing is harvesting your results, they're still using other very clever methods to surface other equally good, yet different results.
I'm stunned that anyone could think that the core issue in this controversy is indexing or page discovery. It should have been totally obvious from the examples in the initial article and blog post that this was specifically about copying ranking.
Oh please. At worst, Microsoft's customers are sharing results of their browsing activities, based on information you gave them. How in the world could that be illegal?
Your team is really coming across as whiny on this issue, which seems to me precisely the wrong tactic to take in response to Bing's increasing market share. It makes your team look defensive, as if your advantage is slipping away and this sort of theatrical display is all you've got in response.
Others interpret this situation as Bing being unable to compete effectively except by looking over Google's shoulder. And then denying it when called out with clear-cut evidence.
IMHO - One of your colleagues should have stopped that blog post from being put up.
I work in search quality at Google. I'm busting my ass every day working on fundamental reimaginings of how results get ranked. I'm going to keep doing that regardless of what bing does, because it makes the world a better place, and it's fun.
But, suppose the stuff I'm working on works out, and tomorrow Google shows up with a wholly new set of awesome results. This is very possible, there is a ton of headroom left in search quality, I've seen the experiments myself.
Then after a few weeks of sniffing clicks, Bing comes up with the same set of revolutionary results, but they have no idea how they got there, they have no idea what the evidence is to rank them there, all they know is that people like those results on Google. Is this fair? Is this ethical? Is this even legal?
People in this whole debate have the idea that the user is creating this association between the result and the query, like the user searched through the whole web and came back saying "hey microsoft, check out this great result for this query! I found it! Isn't is awesome?" They aren't. Users click on whatever results you put in front of them, generally starting with the top and working their way down.
Ranking results is not a science with some objective optimal conclusion. Ranking results is fundamentally subjective, and while data-driven, is ultimately an opinion. The user does have some discriminating power in this whole feedback loop, but it's miniscule compared to Google figuring out how to show them that result in the first place. Bing is taking the closest proxy that they can practically acquire for Google's opinion, and using it directly in their ranking.