karlney
https://underthehood.meltwater.com/blog/2022/11/25/how-we-up...
After those changes were made then we haven't had any major issues with heap spikes. But it depends on a lot of factors, how your data looks, how your queries look and what aggregations that are made so its very hard to give some always applicable advice.
They have gotten A LOT better since the 1.x days
Going up to i3en.12xlarge (or equivalent) would probably have worked as well.
But after that the impact of loosing just one node would be too big.
And during that time so many nodes had became slow and unresponsive that another (for us) previously unseen memory leak started to occur.
Nodes kept building up queues of unanswered ping requests on them. And the requests contained our 100Mb large cluster state, so the heaps filled up and evenmore nodes became unresponsive.
And from then on the whole thing turned into a death spiral of doom.
After trying, and failing to get it under control for 48 hours we gave up and rebuilt the whole cluster from scratch, using the snapshots we store on S3.
The recovery took another 90 hours or so. That was not a fun week.
And yes we have had our fair share of pain with the old cluster for sure.
the new version (7.17) is still behaving a lot better so far and feels a lot more predictable.
I don't want to spoil the other blog posts but we managed to solve almost all of our custom use cases without modifying elasticsearch itself. We still have one custom plugin but only to enhance functionality, not for performance and stability reasons.
Doing a quick googling on elasticsearch and ZGC I just found this https://github.com/elastic/elasticsearch/issues/44321 where the official response from elastic is that only G1GC and CMS are tested and supported.
I still think that it will be worth trying Shenandoah or ZGC if/when we ever get down to do some experiments with larger heaps.
Is there any articles/benchmark/notes or anything that you would be willing to share?
We have considered trying out 64g+ heaps for our cluster but we are concerned about very long gc pauses impacting the search performance.
We also have quite a lot of metadata.
But currently we have concluded that it is not worth the added complexity. But that might change again if/when we learn more or get new requirements.
We do change the number of replicas for recent data vs old data. We actually have a 4 tiered approach to how many replicas we use.
The shard allocation api would also be a very helpful addition to this solution. Currently we can sometimes get into situations where elasticsearch actively works against shardonnay and does very 'stupid' moves out of our control that later needs to be 'fixed' by shardonnay in later iterations. We have tried to disable elasticsearchs own allocation algorithm as much as we can but we still want it active as well as a safety net in case something unexpected happens with the hardware for example.
We do have dedicated master nodes. We also will have query and indexing client nodes that coordinates and merges search responses and handles routing etc. But behind that we utilize all data nodes for both query and indexing load. That has worked well so far due to the fact that indexing load is fairly constant. And we scale query load on recent indexes by having a large number of replicas for those.
Our largest problems have always been the vast difference in query complexity that we get. But we are slowly getting on top of that problem now as well. Some details on how that is done can be found in the following blog post
https://underthehood.meltwater.com/blog/2018/09/28/using-mac...