The venn diagram of "people who want more control over their search experience" and "people who are sick up to the back teeth with companies who insist on an email address even to take their product for a trial spin" is pretty much a cross section of an egg, and control over search is the yolk.
Do you think that Duck Duck Go's (relative) success shows that there are still possibilities here?
Reminds me of this story:
http://www.musicthinktank.com/blog/and-if-only-1-of-those-pe...
I only have to be correct for my own use case (with nearly 28 years experience being myself) and google has to be correct for everyone with (at best) 18 years of search results, of which you have to back out what the user is searching for. e.g. If I look for George Washingtons birthday I could look for information about the person george washington, the actual query george washington birthday, where I might find the data wikipedia:george washington, or an event that could contain it preseidents day.
I know both what my query is, and what a successful answer to my query is (at least the format of an acceptable return). Google has to infer that a query like presidents day and then a query like george washington are related and a successful result was the resource: wikipedia, because it contained george washington's birthday.
The point is:
* these examples are contrived. However, a person can be given the tooling to organize their own data & potential data.
* a person will have insight into what they want, almost certainly better than google.
* if they are wrong/the resources hasn't been cached for them, well, the personalized engine can just simply default to google if no results exist and return them
* privacy.
edit: not only this but having every site I visit (if I wanted this and private & personally encrypted) available and searchable is hugely valuable. Google receives takedown requests, resources move, or I forgot how I received them. Caching/downloading/storing video, images and web results and having them remain private and searchable is pretty much reason enough for me to tell you to take my money.,
So people will want to protect their data & privacy, especially companies.
People will want to have a better search experience, which means that things that are on the internet that they want to see will be there. Google is probably the most monitored for takedown requests/deindexing and they control most of the video portals and certainly the window you view the interent through so they are a target to legitimate and illegitimate takedown requests.
edit: they also have built the components and outsourced them. Hadoop/Map Reduce, most of Apache's products, elasticsearch, and another google offering tensorflow can certainly be made into a search engine. Obviously not google scale, but certainly an individual.
If anything, I think people want to have more of the interesting parts of their lives be more public (in other words, they want to be "famous", whether just among their friends or on a bigger stage). They do want many things to be "private" in the sense that they don't want certain people to see certain things online. But I think we're deluding ourselves if we think they care at all about the fact that google is scanning their emails or search history, or that twitter is mining their tweets, or that snapchat is using their location data.
Just read the blog post on optimizing the elasticsearch cluster. It sounds like you have a pretty awesome deployment setup. Now, I know competitive intelligience is "hot" and "trendy", I mean, for sure it isn't mobile-first omni-channel blockchain bitcoin disruptive "trendy", but it's prety close. Now, I jest, but what I am about to say next I am dead serious about: your product is a search engine, just let people use it as a goddamn search engine.
Now, let me explain. For like probably over a year now I have been making the case that between google, elastic and apache there is just a goddamn open source version of google sitting out there. Would a consumer facing pay for search engine be profitable? idk, is your product profitable? I suspect there is a lot of people who would pay a $9.99 a month to upload and control page rank and not have google own them. Lets assume you were able to capture even 0.01% of googles customers e.g. 1 billion people and got them to pay you like $9.99, then you'd have just shy of $1M MRR, and $11.98M annually. Obviously, you could do some nice caching of resources like stack overflow, reddit, hacker news, and wikipedia and simply store peoples preferences (sort of a heirarchical git like system) and then you would have some pretty swiggity sweet optimizations.
You know what might really add value? Having NLP on your own custom tailored information database. I bet people (who search and use the internet daily) would probably see much higher leverage from this than the sentiment analytics you can give them on a competitor. Also, as an aside, you should be mining the competitors customer support channels not only for sentiment but for their inabiliy to service feature requests/complaints. Proabably one of the biggest predictors of why wealthfront lost MS to betterment. Regardless though, if you guys are looking for an expansion strategy or a pivot, this is it.
tl;dr. Peter Theil was wrong (no, not just about Clinkle) but it is true that:
The next larry page & sergei brin will build a search engine.