And one wonders why it is impossible for a new search engine to get started :-) When Google started paying over a billion dollars a year to buy search traffic it was pretty clear that it was not going to be possible to grow a new full stack search engine organically.

Now they are over $4B/year buying traffic, its pretty crazy.

I am not a legal expert, but at what point does the DoJ step and investigate Google for monopolizing 'search'.

When they start abusing it. Just being a monopoly isn't illegal

I believe that when it comes to search, the aim should not be about competing with Google in it's current form. That is impossible. The aim should be to change the architecture of search such that it becomes a commodity that anyone can plug into. Like what open source CMSes did to the AOL/Time Warner giant. Then the owners of this commoditized search can charge for their premium API on a usage basis, transforming search into an ad-free service found anywhere

I think that is an excellent vision but to understand the challenge of achieving it consider how a search engine has to operate. Typically it 'shards' the index of all available documents into chunks, where the necessary descriptive bits for a chunk can fit in the memory of a dense memory server. Then each server returns its "best" result which are collected, and from there the top 10 are selected.

Search needs to be fast. When you click search you want the page to then pop up with your 10 blue links. The physics of light suggest that your index files all need to be close together, both physically and with high bandwidth to support a couple of round trips for distributing the query and returning the results.

While I learned how search worked while I worked at Google, it was a brilliant engineer named Keith Peterson at Blekko (now at Google :-) who taught me some of the more subtle issues involved in balancing indexes, shards, and query processing in order to maximize utilization.

That requirement of closeness and high bandwidth, makes federation hard. Even forwarding a search to Bing (via the Yahoo APIs) to check for additional "long tail" results really challenges your ability to deliver fast results.

Unlike CMSes where they could run independently in parallel, search is a 'talk to everyone all the time' sort of problem, you don't know apriori which indices may have relevant documents (which is why it is called "search" ;-))

Good write up. Thanks

Oh, not much to wonder I reckon. Thanks for the info.

With the amount of data Apple has from Siri I’m constantly surprised they don’t at least have a stab at building a search engine, possibly one with very fine grained privacy/accuracy controls.

I don't think Apple does "fine grained controls"

For the specific context of privacy, they have had more fine-grained controls than Android since the inception of the appstore, until release of Android 6.0 where Android AFAIK has caught up.

Almost, but still not. There’s still no way to revoke android.permission.INTERNET

And if you revoke any permission to any Google apps, all of them will refuse to do shit, and you can’t even place calls without giving Google access to your location, body sensors, bluetooth, etc.

It’s one setting thinking about it: “Track me and make my search results more accurate”.

You have to understand that search engine is not a reflection of what people search. It's about searching the 'web'.

Apple can just acquire DuckDuckGo.

Isn't that just mostly using major search engine indexes? I know DDG some of its own indexing, but how much do they really do on their own? I don't see the benefit of getting DDG for them when they'll have to deal with the major search engines providing most of the data anyway.