It shouldn't have happened in a production environment, but setting up a way to lose tens of thousands of dollars if performance is bad, trusting unreasonable numbers, trying to produce no garbage, and requiring no GC pauses because margins are too tight are all your faults, not a shortcoming of your JVM.
It's my fault that if you trade US equities based on old information you lose money?
Edit: Actually, I'm still not sure how to correctly measure GC pauses, other than from inside the application.
Can you explain how Java was chosen for your trading application? Also, why were you unable to test it properly?
The application was tested in an environment that's a lot like production, except that we had substantially fewer cores than an exchange running the same number of matching engines, so the incoming ticks were a bit less bursty. That burstiness was important in this instance. A big reason for the differences between the production environment and the test environment was the cost of hardware.
No criticism, as I've only read others' accounts, but that sounds out of character.
What's "out of character" (nice euphemism!) is adopting state of the art automatic garbage collection technology, writing in a language that makes actual garbage collection practically unavoidable, and then expecting that garbage collection doesn't happen or happens accordingly to arbitrary expectations. The JVM is clearly inappropriate technology if latency is important: it can perform well in common cases and with a reasonable level of tuning effort, but other options simply do not have the threat of GC pauses.
I'm sorry to say it, but the loss of money here was not the JVMs fault. It was putting code with insufficient load testing in a production environment and crossing your fingers despite the fact that serious money was on the line.
Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.
We did not generally physically swap out servers each time we changed a line of code. Rather, we had two identical sets of servers, one for production and one for test. For test, we had some additional machines to run matching engines. If we wanted to make the test environment more closely match an actual venue, we would need many more machines that run matching engines. That still might not be sufficient to produce identical output timings in prod and test, though, because venues generally do not publish the sequence of all incoming ticks they receive during a day, the source code of their matching engine, what hardware they are using, their kernel version, and so on.
> Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.
It's strange that you replied to me suggesting we had no production-like test environment ~10 hours after I replied to you saying that this code did fine in our production-like test environment.