News for hackers. Stuff that matters.
Re:Ryzen Threadripper Pro 3995WX Spotted by vosper on Friday July 10, 2020 @04:36AM (#23789406)
I’m not sure about “require”, but we use multi-hundred-gigabytes or RAM instances for our Elasticsearch coordinator nodes, which seems to work well.
Re:Ryzen Threadripper Pro 3995WX Spotted by CSDude on Friday July 10, 2020 @04:41AM (#23789442)
Elasticsearch recommends <32GB because of 32-bit per their doc, is it different for coordinator nodes?
Re:Ryzen Threadripper Pro 3995WX Spotted by Xylakant on Friday July 10, 2020 @06:05AM (#23789844)
The underlying reason is that at around 32GB the JVM internally shifts from 32bit pointers for HEAP management to 64bit. The feature is called compressed object pointers (compressed oops) The increased pointer size uses substantial amounts of RAM, a reasonable estimate is that you need at least ~45GB to not have a loss in total available HEAP. Also, GC cycles get longer the more HEAP you have. However, some workloads may require more heap to even be possible and there’s pauseless garbage collector implementations able to handle substantial amounts of memory, so it is possible. It’s just not something you should be doing unless you know exactly what you are doing.

See https://wiki.openjdk.java.net/display/HotSpot/CompressedOops

Re:Ryzen Threadripper Pro 3995WX Spotted by reitzensteinm on Friday July 10, 2020 @06:24AM (#23789956)
Lacking value types, larger pointers will probably penalize JVM programs more than their peers.
Re:Ryzen Threadripper Pro 3995WX Spotted by SomewhatLikely on Friday July 10, 2020 @09:04AM (#23791041)
While there are no JEP 169 style Value Types, you can still layout your data in a way that significantly reduces the number of pointers. For example using parallel arrays of primitives. It's not as clean, but it is one mitigation.
Re:Ryzen Threadripper Pro 3995WX Spotted by behohippy on Friday July 10, 2020 @08:29AM (#23790798)
A "perfect" ES node has 64 gig of memory. 32 to heap and 32 for off heap as well as 8-12 cores for executing queries. Heap is mostly used for internal operations and the inverted index stuff (fielddata), off-heap was used for the columnar data store (docvalues). Most server class machines exceed this nowadays, so we would recommend containerizing multiple instances of ES on a single machine and using the rack awareness feature to make sure shards didn't replicate onto the same physical host. It's been a while since I worked at Elastic, so this info might be out of date by now, but we're still using this formula for our ECE setups where I work now.
Re:Ryzen Threadripper Pro 3995WX Spotted by adamtulinius on Friday July 10, 2020 @05:10AM (#23789573)
That recommendation is only for the JVM heap to my knowledge.
Re:Ryzen Threadripper Pro 3995WX Spotted by Xylakant on Friday July 10, 2020 @06:17AM (#23789919)
For coordinator nodes, only the HEAP is relevant, excess RAM for disk buffers is only of interest for data holding nodes.