Netty is a super complex and also super poorly documented project. I did weeks of exploration and found these JVM args:
-Dio.netty.allocator.numDirectArenas=0
-Dio.netty.noPreferDirect=true
-Dio.netty.noUnsafe=true
work pretty well for us on any HTTP server. They slightly reduce performance as HTTP pool is weaker, but deceases memory usage, also eliminated one of a few memory leaks in an older version of KeyCloak.
https://netty.io/news/2022/11/09/4-1-85-Final.html mentions that a potential memory leak was fixed. In includes a debug log warning of the leak; but enabling debug logging may be a no-no.
Perhaps could be worth reverting that and see if there's any change. Sounds cheap and harmless.