Shameless plug: I'm working on RediSearch, an open-source, in-memory Redis module written in C that does search. Not nearly full-featured as Lucene and friends, of course, but it's a very young project. http://redisearch.io

I'm not really sure what you're trying to say in your comment. Sure, there's a lot of Java going on there, but does that really matter? They're tools and you can interact with them from the language of your choice.

I get the competition part, but none of the above are exactly stagnant, so I'm wondering what you'd like to see more competition achieve.

Not trying to be difficult, just curious in case I missed something from your comment :)

> so I'm wondering what you'd like to see more competition achieve

There's a lot that competition can help improve, for example in the areas of performance, robustness, and also in functionality, e.g. better NLP for better understanding of queries and translating them into results, image/audio search, etc. And competition can also come up with surprising new features that we can't even think of right now.

This, plus it is (imho) quite weird that we have only one source of code for one of the basic branches of Computer Science.

Take a look at http://bitfunnel.org/, an open-source version of some of the algorithms from Bing. It's not ready for production use but you may enjoy following along in the implementation of a large search engine. The design notes and blog are also worth reading.

IMO this is like saying Hadoop or Spark or Kafka needs more competition outside the Java (JVM) world. Elasticsearch, at least, has a dead simple web API which makes it accessible from any platform, along with a slew of clients and integrations with other tools/platforms.

This. ElasticSearch is one of only two pieces of Java (the other one being Flyway) that's so good we use it anyway. Not too bad anymore with Docker, it's a REST API after all.

In the end, ES really proved to be the least bad search server there's out there. The real crux isn't search, it's language. And as Lucene is made by technical linguists (a really rather special bunch) and Java is still universities' darling, it's unlikely their effort can be redone in a non-JVM language anytime soon.

Have you looked at OkLog?

I've used Sphinx (http://sphinxsearch.com) in a previous job. If it fits your needs then it's pretty nice.

Why do you care that those black boxes are implemented in Java?

There's two main reasons I can see: performance is impacted by the implementation language and because officially supported SDKs/client libraries are usually written, and get updates, in the implementation's language first e.g. Elasticsearch and Java.