Java BitSet performance-flicker mystery

Re:Java BitSet performance-flicker mystery by CraigJPerry on Sunday February 06, 2022 @07:32AM (#30231945)

My gut reaction is that this[1] looks like a NUMA issue. The flip-flop performance profile immediately reminds me of a very gnarly prod issue i lead the investigation of. I’d want to eliminate NUMA first from my investigation.

You can use “numactl --cpunodebind=0 --membind=0 <your original java benchmark command>” to pin the JVM to a particular NUMA node and immediately discount my notion as bunkum.

[1] all the reasons mentioned that make me think numa:

  1. The machine affected has multiple NUMA domains
  2. The M1 is a single NUMA domain chip and does not exhibit the behaviour 
  3. The real thing making me think this - the consistent performance flip flop is close to the ~40% performance overhead of accessing remote memory vs local that i’ve seen previously. You’re seeing higher overheads but your code is a much tighter loop of memory access than my previous experience so that could explain the difference i think.

[ Parent | on HN ]

Re:Java BitSet performance-flicker mystery by cospaia on Sunday February 06, 2022 @11:25AM (#30233963)

This seems to be the issue. Running it 10 times gives me quite consistent performance near the top.

    0 - Passes: 5876, count: 78498, Valid: true
    1 - Passes: 5870, count: 78498, Valid: true
    2 - Passes: 5944, count: 78498, Valid: true
    3 - Passes: 5903, count: 78498, Valid: true
    4 - Passes: 6051, count: 78498, Valid: true
    5 - Passes: 5874, count: 78498, Valid: true
    6 - Passes: 5902, count: 78498, Valid: true
    7 - Passes: 6041, count: 78498, Valid: true
    8 - Passes: 5906, count: 78498, Valid: true
    9 - Passes: 5889, count: 78498, Valid: true

Will run it 100 times now to lower the risk that this is just a coincidence.

[ Parent | on HN ]

Re:Java BitSet performance-flicker mystery by CraigJPerry on Sunday February 06, 2022 @01:15PM (#30235224)

Ahh nice!

The thing that blew my mind when i ran into this was that the jvm had actually been NUMA aware for many years at that point but it was all turned off by default. This was jdk8 days so things are probably better now.

After i figured the root cause, I spent about a day logging results in excel as i tested various GC options, the only saving grace was i recall the Oracle documentation for GC options and their NUMA effects was pretty good. That optimisation time was really well spent though, i got meaningful improvements in both service latency and throughput.

[ Parent | on HN ]

Re:Java BitSet performance-flicker mystery by cospaia on Sunday February 06, 2022 @06:49PM (#30238363)

I had this issue using JDK17 and 18 as well, so at least by default things are not better. =)

I wonder if there is a way that does not involve using `numactl`. It needs the container to run in privileged mode and that is not how the containers are run in the drag-racing benchmark...

[ Parent | on HN ]

Re:Java BitSet performance-flicker mystery by CraigJPerry on Sunday February 06, 2022 @07:52PM (#30238790)

Can you use the jvm’s -xx:+UseNUMA option instead?

[ Parent | on HN ]

Re:Java BitSet performance-flicker mystery by cospaia on Tuesday February 08, 2022 @04:08PM (#30264730)

Thanks! It does and it does not. Here's 100 rounds where it does: https://github.com/PEZ/ghost-chase-condition/tree/master/tes...

But in other cases I have it seems to not help that much. Not sure how to minimize those, but will try figure it out.

[ Parent | on HN ]