You can use native memory tracking to better inspect offheap usages. I had a similar problem where the default MaxMetaspaceSize was excessive and adjusted it to a reasonable setting.
-XX:NativeMemoryTracking=summary
jcmd <pid> VM.native_memory summary
So in my above scenario, I could indeed run the app with a -Xmx32M and yet the application was still using 200-300MB of resident memory (RSS) due to offheap usage.