News for hackers. Stuff that matters.
Re:Native Minecraft servers with GraalVM Native Image by Thaxll on Friday September 02, 2022 @09:43AM (#32690704)
"As such, it is supposed to require fewer CPU and memory resources, provide better startup times, and be easier and cheaper to deploy."

So we don't even know if it actually makes things faster? Startup are a none issue, CPU / memory is but you need proof for that.

Graal does not support ZGC or Shenandoah so it's hard to say if the G1 version from Graal is up to speed.

Re:Native Minecraft servers with GraalVM Native Image by fniephaus on Friday September 02, 2022 @10:00AM (#32690969)
Disclaimer: I work on the GraalVM team.

The students "measured noticeable reductions in terms of memory footprint of up to 43%" [1] in some preliminary experiments. More from the accompanying blog post:

"We also hope that the Minecraft community builds on our work and helps benchmark different configurations for native Minecraft servers in more detail and in larger settings."

Please feel free to share any numbers on CPU/memory usage with us!

[1] https://medium.com/graalvm/native-minecraft-servers-with-gra...

Re:Native Minecraft servers with GraalVM Native Image by pwagland on Friday September 02, 2022 @11:18AM (#32691829)
Note that the memory usage _could_ potentially be significantly improved for the JVM by just using an alternative allocator, such as jemalloc. In our system, we saw, in some instances, native memory usage decrease by about 60%, and it also resolved a slow "leak" that we saw, since glibc was allocating memory, and not returning it to the OS. In our case it was because we were opening a lot of class loaders, and hence zip files, from different threads.
Re:Native Minecraft servers with GraalVM Native Image by xendo on Friday September 02, 2022 @04:37PM (#32695462)
I can second what you wrote about jemalloc. Some internal services at Amazon are using it with solid outcomes. I also recommend trying out 5.3.0 version released earlier this year.
Re:Native Minecraft servers with GraalVM Native Image by OniBait on Friday September 02, 2022 @06:48PM (#32696584)
Last I did benchmarking, a vast majority of memory allocations were strings that were typically all dereferenced right away and cleaned up in GEN1 GC. I had contemplated whether string pooling would be useful or not but never got around to it. Would be interesting to see if you could get reduced memory usage and potentially better performance by decreasing pressure on the GC during the GEN1 phase.

(Side note: this was when I was co-maintaining MCPC so was typically with mods installed and they heavily use NBT which I suspect is where a lot of that string allocation was happening.)

Re:Native Minecraft servers with GraalVM Native Image by bitcharmer on Friday September 02, 2022 @11:59AM (#32692341)
This is very interesting. Could you share more details on this particular issue in glibc? Jar files get mapped so I'm really interested where glibc failed to release memory.
Re:Native Minecraft servers with GraalVM Native Image by xendo on Friday September 02, 2022 @04:41PM (#32695486)
No the OP, but we had similar issue — our service was leaking when allocating native memory using JNI. We onboarded Jemalloc as it has better debugging capabilities, but the leak dissapeared and performance improved. We never got around to root causing original leak.
Re:Native Minecraft servers with GraalVM Native Image by Birch-san on Friday September 02, 2022 @09:52PM (#32697798)
It's probably the same thing prestodb encountered: https://github.com/prestodb/presto/issues/8993
Re:Native Minecraft servers with GraalVM Native Image by Birch-san on Friday September 02, 2022 @09:51PM (#32697795)
For performance reasons, glibc may not return freed memory to the OS. You can increase the incentive for it to do so, by reducing MALLOC_ARENA_MAX to 2. https://github.com/prestodb/presto/issues/8993
Re:Native Minecraft servers with GraalVM Native Image by cogman10 on Friday September 02, 2022 @05:26PM (#32695887)
I was under the impression that most builds of the JVM used jemalloc by default.
Re:Native Minecraft servers with GraalVM Native Image by solarkraft on Friday September 02, 2022 @10:33AM (#32691311)
Why is this? I thought the JVM already did somewhat decent JIT compilation ...

If I understand the article correctly, you're preempting all possibly unoptimized/expensive code paths (reflection) by attempting to literally execute all of them? While it's a cool experiment, isn't it a bit error-prone (besides being a lot of effort of course, but playing Minecraft on the side does sound pretty fun!)?

Re:Native Minecraft servers with GraalVM Native Image by kaba0 on Friday September 02, 2022 @10:41AM (#32691398)
The JVM is likely to beat AOT compiled java code in almost all cases - but due to Graal having a closed-world assumption (e.g. no unknown class can be loaded, so a non-final class knows that it won’t be overridden allowing for better optimizations, limited reflection allows for storing less metadata on classes, etc) it does allow for significant memory reduction. Also, escape analysis is easier in an offline manner.
Re:Native Minecraft servers with GraalVM Native Image by pvillano on Saturday September 03, 2022 @07:46PM (#32707752)
can't that all be done speculatively with de-optimization /s
Re:Native Minecraft servers with GraalVM Native Image by fniephaus on Friday September 02, 2022 @10:41AM (#32691403)
JIT compilation requires additional CPU and memory resources at run-time, which AOT compilation can avoid. This also means that for a native executable, the compilation work only needs to be done once at build-time and not per process.
Re:Native Minecraft servers with GraalVM Native Image by bitcharmer on Friday September 02, 2022 @12:02PM (#32692374)
This is the first time I see someone bring up extra cpu and memory usage as a downside of JIT. It might matter in the embedded world but it's Java we're talking about so the cost is minuscule compared to what you're getting for it.
Re:Native Minecraft servers with GraalVM Native Image by salmo on Friday September 02, 2022 @09:48PM (#32697770)
You’re not wrong, but it is funny how we got here from Gosling’s Oak addressing set top boxes.

The thing was built to address the burgeoning embedded w/ a little horsepower market with its variety of hardware and OSes.

Now it runs Enterprise server software… and Minecraft.

Re:Native Minecraft servers with GraalVM Native Image by kaba0 on Saturday September 03, 2022 @03:47AM (#32699688)
Well, it does make sense - a controlled runtime failure is much better than a segfault, or worse, a silent failure corrupting heap. Pair it with decent performance even back than, increased developer productivity and the best observability tools, which is again helped by the VM-semantics.
Re:Native Minecraft servers with GraalVM Native Image by cogman10 on Friday September 02, 2022 @05:32PM (#32695948)
Those are usually pretty trivial as they are judiciously handed out based on hot code paths by the JVM.

There are certainly pathological cases where it could cause major issues.

AOT suffers from not having runtime information, so anything involving dynamic dispatch (which is REALLY heavily used in java) will be a lot harder to optimize. JITs get to cheat because they know that the `void foo(Collection bar)` method is always or usually called with an `ArrayList`. PGO is the AOT world's answer to this problem, but it generally explodes build times and requires real world usage.

In java land, there's also the option of "AppCDS" which can cut down a large portion of that compilation time between processes.

Re:Native Minecraft servers with GraalVM Native Image by doikor on Friday September 02, 2022 @03:39PM (#32694941)
GraalVM does have a better optimizer in certain conditions than C2 in vanilla JDK which can can lead to better performance. Basically the only way to know if GraalVM will give you better performance or not is to try it and/or run benchmark your code.

https://www.graalvm.org/22.2/examples/java-performance-examp...

Re:Native Minecraft servers with GraalVM Native Image by zamalek on Friday September 02, 2022 @01:25PM (#32693386)
Is there any benefit to simply running/JIT the client and server on GraalVM instead of JVM?
Re:Native Minecraft servers with GraalVM Native Image by spullara on Friday September 02, 2022 @07:43PM (#32697003)
It is much worse than this because the free version of graalvm only supports the serial garbage collector. Minecraft servers and clients should be using ZGC to get rid of garbage collection pauses.
Re:Native Minecraft servers with GraalVM Native Image by Kukumber on Friday September 02, 2022 @07:58PM (#32697094)
startup time is SUPER important

lets you spawn new game instances on the fly, reduce time spent loading the chunks and game data

you save a lot of money when you scale, and you improve latency, people don't complain with huge loading time and stutters for fresh servers

ask anybody working on the industry

fun fact, that's the first thing Riot did when they acquired Hytale

They rewrote their C# client to C++ for portability

And they rewrote their Java server to C++ for performance (and cost saving)

Tech Change: https://hytale.com/news/2022/7/summer-2022-development-updat...

Re:Native Minecraft servers with GraalVM Native Image by vips7L on Friday September 02, 2022 @12:38PM (#32692814)
Are they using Graal enterprise? Last I checked Community Edition of native image uses the serial collector not G1.
Re:Native Minecraft servers with GraalVM Native Image by fniephaus on Saturday September 03, 2022 @01:47AM (#32699110)
The students used both, the community and enterprise editions of GraalVM. Indeed, G1 is an enterprise feature: https://www.graalvm.org/22.2/reference-manual/native-image/o...
Re:Native Minecraft servers with GraalVM Native Image by majidazimi on Friday September 02, 2022 @09:58AM (#32690936)
>Startup are a none issue

Yes it is. Developing any short term job -- that runs multiple seconds and goes away -- like lambda or k8s jobs with Java is meaningless for exactly this reason. The startup time is longer than the run time.

Re:Native Minecraft servers with GraalVM Native Image by HideousKojima on Friday September 02, 2022 @09:59AM (#32690951)
A Minecraft server is not a short term job.
Re:Native Minecraft servers with GraalVM Native Image by vgalin on Friday September 02, 2022 @10:28AM (#32691261)
I guess it can be in a specific case: minigames servers (such as Hypixel), which are just a bunch of servers "connected" together. Players start into a "lobby" server, where they can choose a minigame, and are then sent to another server where they spend a few minutes.
Re:Native Minecraft servers with GraalVM Native Image by curtainsforus on Friday September 02, 2022 @10:33AM (#32691304)
The game servers don't restart after the end of a round, though, do they? I'd imagine they kick the players back to the lobby, reset the in-server game, and then tell the lobby to send the next batch of players.
Re:Native Minecraft servers with GraalVM Native Image by terafo on Friday September 02, 2022 @10:41AM (#32691394)
You assume that load is constant, it isn't. And load varies not only with amount of players on minigames server, but with changes in distribution of players between minigames also.
Re:Native Minecraft servers with GraalVM Native Image by curtainsforus on Friday September 02, 2022 @01:48PM (#32693637)
There's usually more than one server per minigame. You could see it in the url you were redirected to; they had more servers running the more popular minigames. Each minigame has a player limit, so the maximum load on any given minigame server is known (within the bounds of the minecraft sub-superset that makes up that minigame- but usually the minigames are deliberately limited/bounded in how much computation they need, as opposed to vanilla Minecraft). Extra players get sent to the next available server. If there's consistent overflow, at that point you might turn on a whole new server, or change a server's gamemode (I don't know to what degree Hypixel actually did/does this, or how often it's actually necessary).
Re:Native Minecraft servers with GraalVM Native Image by solarkraft on Friday September 02, 2022 @10:40AM (#32691387)
This is fairly certainly how it used to be done. Besides, you can have the server idle a good bit before actually letting players in.
Re:Native Minecraft servers with GraalVM Native Image by kaba0 on Friday September 02, 2022 @10:42AM (#32691412)
The JVM can start up in less than 0.1 seconds. Depending on the amount of classes being loaded it is not an issue even for lambda and k8s jobs.
Re:Native Minecraft servers with GraalVM Native Image by vips7L on Friday September 02, 2022 @12:41PM (#32692858)
The VM starts up plenty fast. The slow part is when people use reflective dependency injection containers that take seconds to scan the class path before before executing.
Re:Native Minecraft servers with GraalVM Native Image by imtringued on Friday September 02, 2022 @01:48PM (#32693652)
This is why frameworks like Micronaut exist.
Re:Native Minecraft servers with GraalVM Native Image by vips7L on Friday September 02, 2022 @02:56PM (#32694511)
Micronaut, Quarkus, Avaje Inject, CDI Lite. Plenty of solutions if people would stop reaching for Spring.
Re:Native Minecraft servers with GraalVM Native Image by CSDude on Friday September 02, 2022 @11:00AM (#32691630)
You clearly did not deploy enough classes on Lambda to have more than 10 seconds warmup on a trivial Java based Lambda function.
Re:Native Minecraft servers with GraalVM Native Image by bitcharmer on Friday September 02, 2022 @12:03PM (#32692393)
Warm-up is more a function of invocation count rather than time as you seem to be suggesting here
Re:Native Minecraft servers with GraalVM Native Image by antome on Friday September 02, 2022 @10:01AM (#32690980)
This is a Minecraft server, so it's going to be running 24/7.
Re:Native Minecraft servers with GraalVM Native Image by terafo on Friday September 02, 2022 @10:35AM (#32691329)
I see you aren't familiar with modern state of Minecraft servers. Due to Minecraft being limited to a one core big servers actually aren't a single instance. They use proxy servers(such as BungeeCord and it's forks) which distributes load between several lobby servers and from there people join one of custom gamemodes(Skyblock, Bedwars, etc). This allows for tens of thousands of people to play simultaneously, but not in the single world, while SMP(Survival Multiplayer) servers can run couple of hundreds at most. These giant servers are heavily containerized and automatically scale under load, so spinning up and shutting down servers is a pretty normal thing. And there have been some attempts to make Minecraft to run a single world on multiple instances(MultiPaper and some private ones), so even for usual SMP server it can be a commonplace soon as player join and leave.
Re:Native Minecraft servers with GraalVM Native Image by doodlesdev on Friday September 02, 2022 @11:02AM (#32691668)
> And there have been some attempts to make Minecraft to run a single world on multiple instances(MultiPaper and some private ones)

First time I hear about MultiPaper, another idea I had which I din't know someone was already working on LOL. It's a pretty promising idea considering the current performance problems of the game. This could possibly allow thousands of players in the same server which would be AMAZING, almost a completely different game. Imagine if MultiPaper was compiled to native using GraalVM.

Re:Native Minecraft servers with GraalVM Native Image by mcculley on Friday September 02, 2022 @10:25AM (#32691224)
I encourage my competitors to keep thinking this.