ES is not this way, we have to manage our nodes ourselves and figure out "that one node is failing, why?" type questions. I hear they're working on a serverless version, but honestly, I think we will be leaving ES before that happens.
We have a similar experience except we’re using AWS’ version, so any inquiry as to what happened ends with support saying “maybe if you upgrade your nodes this won’t happen again? idk”
The replication protocols and leader election were IMO not battle-hardened or likely to pass Aphyr-style testing. It was pretty easy to get into a state where the source of truth was unclear.
Source: Ran an Elasticsearch hosting company in the 2010's. A little out of the loop, but not sure much has changed.
To make generational GC efficient, you want to have very short lived objects, or objects that live forever. Lots of moderately long-lived objects is the worst case scenario, as it causes permanent fragmentation of the old GC generation.
Lucene generally churns through a lot of strings while indexing, but it also builds a lot of caches that live on-heap for a few minutes. Because the minor GCs come fast and furious due to indexing, that means you have caches that last just long enough to get evicted into the old generation, only to become useless shortly thereafter.
The end result looks like a slow burning memory leak. I've seen the worst cases take down servers every hour or two, but this can accumulate over time on a slower fuse as well.
Tuning the older options wasn't really all that beneficial. We tried tuning, then running a custom build of OpenJDK to mess with the survivor space (which isn't that tunable via config), then ultimately settled on more aggressive (i.e. weekly) rolling restarts of servers.
Additionally, many companies learn the hard way that dynamic mapping is a bad idea because you might end up with hundreds of fields and a lot of memory overhead and garbage collection. I've fixed more than a few situations like this for clients that ended up with hundreds or thousands of fields, many shards and indices. Usually it's because they are just dumping a lot of data in there without thinking about how to optimize that for what they need.
A properly architected setup is not going to fall over randomly. But you need to know what you are doing and there are a lot of clients that I help that clearly don't have the in house expertise to do this properly and are a bit out of their depth.
https://www.elastic.co/guide/en/elasticsearch/resiliency/mas...
>7 its become quite reliable. It comes down to how you manage and maintain the cluster
The main issue is not lack of robustness but the fact that there are no transactions and it doesn't scale very well on the write side unless you use bulk inserts. You can work around some of the limitations with things like optimistic locking to guarantee that you aren't overwriting your own writes. Doing that requires a bit of boiler plate. For applications with low amounts of writes, it's actually not horrible if you do that. Otherwise, if you make sure you do regular backups (with snapshots), you should be fine. Additionally, you need to size your cluster properly. Things get bad when you run low on memory or disk. But that's true for any database.
If you are interested in doing this, my open source kotlin client for Elasticsearch and Opensearch (I support both) has an IndexRepository class which makes all this very easy to use and encapsulates a lot of the trickery and bookkeeping you need to do to get optimistic locking. I also have a very easy to use way to do bulk indexing that takes care of all the bookkeeping, retries, etc. without requiring a lot of boiler plate. And of course because this is Kotlin, it comes with nice to use kotlin DSLs.
You can emulate what it does with other clients for other languages but I'm not really aware of a lot of other projects that make this as easy. Frankly, most clients are a bit bare bones and require a lot of boiler plate. Getting rid of boiler plate was the key motivation for me to create this library.
I’ll admit haven’t kept up with this but is it still the case that Elasticsearch is “not a reliable data store”?
I remember there used to be a line in the Elasticsearch docs saying that Elasticseach shouldn’t be your primary data store or something to that effect. At some point they removed that verbiage, seemingly indicating more confidence in their reliability but I still hear people sticking with the previous guidance.