They should throw it away and build something new.

I am no fan of a lot of the design choices or the effort required to operate large ES clusters, but there are easily several hundred man-years of development effort behind ES. It would be a really long time before a team starting from scratch could produce something better.

Check out some of the things going on in the Rust space MeiliSearch and QuickWit are 2 options emerging (targeting slightly different use cases), and Tantivy -the Rust equivalent of Lucene- is looking really good.

Quickwit CEO here. For people interested, we are focusing on immutable data (logs of all kinds typically). https://github.com/quickwit-oss/quickwit

We make it much 10x cheaper and easier to scale and operate. We store the index on Amazon S3.

I can’t say I disagree. Lucene is a beautiful, scientific thing; but ES is an increasingly-complex mess of opaque API’s and broken promises. Search engines are really, really hard. There is so much room for something better, more full-featured, more tunable, more reasonable.

How is it a mess?

Feature bloat, bad documentation, API models that are nearly inscrutable. Backwards-incompatible changes are constant, requiring reindexing.

I’d like to see the architecture re-thought to include an indexer, like Apache Druid. I’d like to see a proper command line tool (‘searchctl’?), and API clients a-la gRPC (to be less Java-centric). I’d generally like the project to go back to basics and design for 2022 applications.

On a related note, Druid is hardly a good example of API and good deployment and docs story.