When you have an expectation from the exchange that you respond in under 80ms, and 25-50ms of that is eaten up by transport, you don't have a lot of time to mess around.

So you spend the first chunk of time optimizing I/O and how you're accessing it.

Then you start looking at computation -- improving caching, etc.

At a certain point you start noticing in your graphs that you're on average doing quite well. But there's those hiccups every Nth request...

And now you're in the game of fighting with your language's garbage collection algorithm...

Try to improve allocations where possible. There's often plenty of small allocations happening in Java that can be avoided (string usage is one major driver). Less allocations means less frequent garbage collections.

Then one hack is to disable the garbage collection. Let the software run up to consume all the 100 GB of memory of the server and crash and restart. There is no pause of garbage collection when there is no garbage collection.

If it's not enough, the last resort is to write native code or switch to C++.

*sync.Pool, arena allocation strategies