Re:Ryzen Threadripper Pro 3995WX Spotted
by CSDude on Friday July 10, 2020 @04:41AM
(#23789442)
Elasticsearch recommends <32GB because of 32-bit per their doc, is it different for coordinator nodes?
[ Parent
| on HN ]
The underlying reason is that at around 32GB the JVM internally shifts from 32bit pointers for HEAP management to 64bit. The feature is called compressed object pointers (compressed oops) The increased pointer size uses substantial amounts of RAM, a reasonable estimate is that you need at least ~45GB to not have a loss in total available HEAP. Also, GC cycles get longer the more HEAP you have.
However, some workloads may require more heap to even be possible and there’s pauseless garbage collector implementations able to handle substantial amounts of memory, so it is possible. It’s just not something you should be doing unless you know exactly what you are doing.
See https://wiki.openjdk.java.net/display/HotSpot/CompressedOops
Re:Ryzen Threadripper Pro 3995WX Spotted
by reitzensteinm on Friday July 10, 2020 @06:24AM
(#23789956)
Lacking value types, larger pointers will probably penalize JVM programs more than their peers.
Re:Ryzen Threadripper Pro 3995WX Spotted
by SomewhatLikely on Friday July 10, 2020 @09:04AM
(#23791041)
While there are no JEP 169 style Value Types, you can still layout your data in a way that significantly reduces the number of pointers. For example using parallel arrays of primitives. It's not as clean, but it is one mitigation.
A "perfect" ES node has 64 gig of memory. 32 to heap and 32 for off heap as well as 8-12 cores for executing queries. Heap is mostly used for internal operations and the inverted index stuff (fielddata), off-heap was used for the columnar data store (docvalues). Most server class machines exceed this nowadays, so we would recommend containerizing multiple instances of ES on a single machine and using the rack awareness feature to make sure shards didn't replicate onto the same physical host. It's been a while since I worked at Elastic, so this info might be out of date by now, but we're still using this formula for our ECE setups where I work now.
Re:Ryzen Threadripper Pro 3995WX Spotted
by adamtulinius on Friday July 10, 2020 @05:10AM
(#23789573)
That recommendation is only for the JVM heap to my knowledge.