My position on the matter might be summarized as "No TCO, no copyable stackful coroutines, no buy." Separately from that a lot of my experience with the JVM involves things like trusting the JVM's claim that it spent 0 microseconds GCing a certain thread that was running code specifically designed to produce no garbage, then running the code in production for one day and losing tens of thousands of dollars, then learning that even if the JVM pauses your thread for 1ms for heap compaction it will still tell you it spent 0 micros on GC.

In other words, on the first day day of testing under realistic load conditions you learned how to measure GC pauses; then, presumably, you were able to improve performance.

It shouldn't have happened in a production environment, but setting up a way to lose tens of thousands of dollars if performance is bad, trusting unreasonable numbers, trying to produce no garbage, and requiring no GC pauses because margins are too tight are all your faults, not a shortcoming of your JVM.

Right, we improved the performance by moving the work out of the JVM.

It's my fault that if you trade US equities based on old information you lose money?

Edit: Actually, I'm still not sure how to correctly measure GC pauses, other than from inside the application.

It's your fault if you trade equities, or you do anything else in which milliseconds matter, with a software platform that is aggressively optimized for average performance and for server stability at the expense of hard real time guarantees and with a language that privileges enforcing constraints (both to ensure correctness and allow optimizations) over allowing the user explicit control. ("Your fault" at the organizational level, not personally, of course.)

Can you explain how Java was chosen for your trading application? Also, why were you unable to test it properly?

The org chose Scala for the trading application in the hope that developers and researchers could both use it, when previously they had been using very different languages. I thought this went mostly well. Scala enables you to use some shenanigans to pretend that a byte-array is an array of structs, and the 31 other threads of the application ran acceptably on the JVM. As an aside, my impression is that Java is somewhat common in the industry.

The application was tested in an environment that's a lot like production, except that we had substantially fewer cores than an exchange running the same number of matching engines, so the incoming ticks were a bit less bursty. That burstiness was important in this instance. A big reason for the differences between the production environment and the test environment was the cost of hardware.

I'm not sure I've ever heard of the JVM being used in trading level software before. I've gotten a broad but vague impression that custom applications are the norm.

No criticism, as I've only read others' accounts, but that sounds out of character.

Java or Clojure applications are "custom" too: custom code running in some kind of application server is more or less the same as custom code relying on libraries and frameworks in C or C++ or other leaner languages.

What's "out of character" (nice euphemism!) is adopting state of the art automatic garbage collection technology, writing in a language that makes actual garbage collection practically unavoidable, and then expecting that garbage collection doesn't happen or happens accordingly to arbitrary expectations. The JVM is clearly inappropriate technology if latency is important: it can perform well in common cases and with a reasonable level of tuning effort, but other options simply do not have the threat of GC pauses.

I gotta agree with HelloNurse here. When that kind of money is on the line you set up a production version that ingests real data at real speed, and produces the same output / response but doesn't actually spend / affect any money. You watch it for at least a week and see what happens.

I'm sorry to say it, but the loss of money here was not the JVMs fault. It was putting code with insufficient load testing in a production environment and crossing your fingers despite the fact that serious money was on the line.

A complete-fidelity production-like environment would be a couple orders of magnitude more expensive than the 1-day loss, so I disagree.

You don't need another complete set of servers: the new production servers, before beginning actual service, are available for brutal stress tests and all sorts of other experiments while the old servers running the old software are still in use and undisturbed. If the system is completely new and there is no old software on old servers, it's even easier.

Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.

> You don't need another complete set of servers: the new production servers, before beginning actual service, are available for brutal stress tests and all sorts of other experiments while the old servers running the old software are still in use and undisturbed. If the system is completely new and there is no old software on old servers, it's even easier.

We did not generally physically swap out servers each time we changed a line of code. Rather, we had two identical sets of servers, one for production and one for test. For test, we had some additional machines to run matching engines. If we wanted to make the test environment more closely match an actual venue, we would need many more machines that run matching engines. That still might not be sufficient to produce identical output timings in prod and test, though, because venues generally do not publish the sequence of all incoming ticks they receive during a day, the source code of their matching engine, what hardware they are using, their kernel version, and so on.

> Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.

It's strange that you replied to me suggesting we had no production-like test environment ~10 hours after I replied to you saying that this code did fine in our production-like test environment.

Do you know of a production ready VM with these properties that has a nice ecosystem?

I reckon part of the point is that a VM (in the JVM or CLR sense) might be a poor fit for the given use case, as might an automatic garbage collector, and thus that might dictate the use of languages which rely on neither (like C/C++ or Rust).

If you do a trivial amount of up-front work, C and Luajit have these properties, but Luajit often isn't a suitable replacement for the JVM and C isn't a VM. So no.

I did a JVM targeting language as a research project, and was involved into porting KVM to an obscure architecture. It's been a while of course (early 00s) so maybe I misremember something or there were huge strides of progress in JVM.

Can you jump over 64Kb in bytecode yet? Allocate a lexically scoped variable?

Of course, enormous amount of work has been put into making this particular pig fly at reasonable speed. Doesn't change the fact that most usability complaints about Clojure stem from reliance on JVM piggybacking.

so what's better and out there and reasonable stable?

Probably BeamVM, giving you better options for scaling, abstracting away machine borders and lightweight processes.

BeamVM is much worse than the JVM in the raw computing performances department.

As far as I know, people in the Erlang world tend to use NIFs for computation heavy tasks: http://erlang.org/doc/tutorial/nif.html However, I have not done that myself, so I cannot tell from experience how that works.

x86-64

market share is going to ARM

ARM is nice too, I concur

From the narrow scope of web development, JVM is fine. Kotlin is able to have every ergonomic feature you'd want from a language, and it gets closer to the performance of C in TechEmpower than many other languages (notably C#, Node, and Python, three of the most popular languages).

Tuning can be a pain, but scaling anything is hard.

It is clear that a lot of very important and practical work had been delivered with Java/JVM. But the original point was not about what is possible, but what is optimal.

Kotlin, from the little I've seen, appears to be largely a syntactic variation of Java, so there is certainly good impedance to JVM. Clojure is less so, and here lies the reason everyone curses about its backtraces.

Java is antiquated though, I don't see many new big libraries/frameworks being written in it. Java is too big and too entrenched in the enterprise to go away any time soon but in ten years the language will become legacy. For what's being used, languages like Go and to some extent Rust are eating Java's market share for new stuff being built. More and more universities are moving away from Java as teaching language too.

Clojure is in a bad spot here, it was a good idea at the time but not today. Clojure relies on Java for its libraries cause Clojure has almost no ecosystem. 95% of Clojure libs are Java wrappers.

Add the various JVM implementations, ART and the .NET variants into it as well.

HN has a very large bro/cowboy coder constituency. I suspect a large portion of the commenters here have not built anything non-trivial. You learn to filter it out but it's disappointing to learn that "engineers" are not much more rational and objective than the average Joe.