News for hackers. Stuff that matters.
Re:Kafka Removing Zookeeper Dependency by elric on Sunday May 17, 2020 @01:45AM (#23209861)
Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question.

You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them shifts some of the complexity of dealing with certificates from the developer to the system administrator. It's an implementation choice.

The only other complaint of yours I could find ITT was about setting min/max heap size... not only is it extremely convenient to be able to do that, it also hasn't ever technically been required, and the defaults have been Good Enough for most uses since Java 5 came out in 2004. The JVM will figure it out for you, and if it's too conservative or too aggressive for you, you can tune it.

If Kafka is a PITA to deploy, fine, maybe that's fair criticism. Not all things on the JVM are pain to deploy.

Re:Kafka Removing Zookeeper Dependency by axaxs on Sunday May 17, 2020 @02:26AM (#23210012)
I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.
Re:Kafka Removing Zookeeper Dependency by elric on Sunday May 17, 2020 @06:27AM (#23211012)
Seriously? When was the last time you deployed something as a standalone, static binary? Hardly anything with any amount of complexity is shipped like that, especially not cross-platform. Just about everything requires some kind of runtime, some kind of dependencies or some kind of environment. I can go "dnf install firefox" just as easily as I can go "dnf install ant". Both of these come with a boatload of dependencies. One of them depends on a JVM, the other doesn't. Both are packaged pretty well and both can be installed in a single step.

I indeed won't admit that "Java apps" are a pain to deploy, because it's a meaningless and overly generalistic statement. Some apps are a pain to deploy, but it's almost always down to the way they're packaged/distributed, and rarely down to the underlying technology.

Re:Kafka Removing Zookeeper Dependency by bsaul on Sunday May 17, 2020 @07:22AM (#23211223)
I'm not sure what you mean. Deploying static binaries was the main way to go before, and is now the standard approach with go. Pretty much every service developed in go is deployed as a standalone statically compiled binary, and i would assume it's the same for code developped in Rust.
Re:Kafka Removing Zookeeper Dependency by Tsarbomb on Sunday May 17, 2020 @09:21AM (#23211784)
Except you need to make sure you built against the correct version of libc because you cannot statically link that.

I remember when docker first came out and people were losing their minds about how they will be able to deploy their applications with all their dependencies in a single shippable bundle, kind of like an uber jar.

Re:Kafka Removing Zookeeper Dependency by nitsky on Sunday May 17, 2020 @09:50AM (#23211962)
Rust programs can be statically linked to musl libc, producing binaries that will run on any linux distribution. You can run ldd on the binaries and confirm they have zero dependencies on shared libraries.
Re:Kafka Removing Zookeeper Dependency by touisteur on Sunday May 17, 2020 @09:46AM (#23211941)
Isn't that what go is trying to alleviate, with their use of direct syscalls and no use of libc?
Re:Kafka Removing Zookeeper Dependency by hkeide on Sunday May 24, 2020 @04:50AM (#23290173)
I build my Linux Go binaries on my macOS development machine. Cross compilation is trivially easy in Go and I never had any libc version problems.
Re:Kafka Removing Zookeeper Dependency by kitotik on Sunday May 17, 2020 @10:02AM (#23212047)
> Hardly anything with any amount of complexity is shipped like that

The entire HashiCorp stack and Cockroachdb comes to mind. I’m sure there are things that are much more complex in the world, but they do some fairly heavy lifting.

> especially not cross-platform

I find it drastically simpler to download a single binary for each platform I need to deploy to(macOS,Linux,FreeBSD) than it is to get a consistent Java environment setup on those same platforms.

Re:Kafka Removing Zookeeper Dependency by tamale on Sunday May 17, 2020 @11:43AM (#23212723)
You're kinda omitting the insane amount of configuration you can give to pretty much every hashicorp product and cockroachdb

I think the big difference is that Java projects TEND to be more 'up front' about their configuration options, with shipped default files with a lot of the options already set to some default value, while projects like the ones you mentioned require you to look up every property yourself and set it to something if you want to change it from the default.

Re:Kafka Removing Zookeeper Dependency by kitotik on Sunday May 17, 2020 @03:06PM (#23214407)
I did not mention the configs, because those are plain text and do not really add dependencies I don’t already have.

Are you saying that Java projects tend to have more sane defaults and come bundled with more out of the box?

The last Java project I had to deploy was ElasticSearch/Kibana, and there was a LOT of configuration needed that required consulting a lot of disparate documentation.

Re:Kafka Removing Zookeeper Dependency by parasubvert on Monday May 18, 2020 @07:45AM (#23221131)
Nearly everything written in Golang is deployed as a static binary with minimal dependencies, which is partly why it has grown in popularity.

It’s a nearly universal experience that different language runtimes have very different experiences with dependencies. Someone who ships a Node, Python or Ruby-based tool for example has a tough time, to the point of needing to write a wrapper installer that vendors all dependencies including the runtime itself, just to be sure.

The JVM with Maven and fat JARs isn’t probably as bad as these cases, as mostly it’s just requiring a compatible JRE. This is why Spring Boot apps for example are so popular for deployment - no more app server insanity!

Jlink makes this more like Golang, but isn’t used enough.

That said Golang dependency management for the developer / builder is a history of horrors. Maven has had its issues but has gotten past most of them.

Re:Kafka Removing Zookeeper Dependency by buzzkillington on Sunday May 17, 2020 @09:27AM (#23211819)
>Hardly anything with any amount of complexity is shipped like that, especially not cross-platform.

Including the JVM. The paths alone.

Re:Kafka Removing Zookeeper Dependency by apta on Sunday May 17, 2020 @11:08AM (#23212511)
You can ship the JVM with your code (uber jar), so no paths to set up. And now with jlink, you can even strip it down to only parts of the JVM that your code uses, to optimize size.
Re:Kafka Removing Zookeeper Dependency by lostapathy on Sunday May 17, 2020 @12:28PM (#23213043)
From the perspective of a java developer, this makes total sense.

From the perspective of a sysadmin who just wants to deploy an app that ships as a jar and doesn't come pre-packaged with any convenience features like you mentioned, it's an awful lot to work out and learn.

Re:Kafka Removing Zookeeper Dependency by stingraycharles on Sunday May 17, 2020 @06:34AM (#23211034)
The problem I have with your statement is that you do not elaborate on why it is such a PITA. Maybe you had some issues with a Spring web app? JavaBeans? I don’t know. The problem is that you’re making a broad generalization about all apps on the JVM, while to me it seems like this has more to do with decisions of the specific apps you were deploying.

To go back to the original argument, why is Kafka ported from JVM to Go suddenly much easier to deploy? Is it really due to the JVM, or is it perhaps related to design decisions made while porting the application to Go?

Re:Kafka Removing Zookeeper Dependency by tiew9Vii on Sunday May 17, 2020 @03:38AM (#23210371)
They are really simple if done correctly. You have a single fat jar so can do java -jar myapp.jar which is almost as easy as your static binary but prefixed with two extra words.

It’s also trivially easy to package as a docker image.

What makes things like Zookeeper and Kafka to deploy is they are complex applications. You can write them in any other language and the deployment will still remain as non trivial.

Re:Kafka Removing Zookeeper Dependency by michaelt on Sunday May 17, 2020 @04:18AM (#23210541)
I use Java for a living. Whether you think Java is very easy or a pain in the ass depends on what you're doing.

Consider the fact Ubuntu 18.04's default repos come with a pretty ancient version of Java. And if you want a newer version, you have to use some third-party source. And the fact there are both OpenJDK and Oracle JDKs to choose from. Will that be JDK or JRE? Headless? Depending on the combination of JDK and software, maybe the fonts will go all funny. Got multiple JDKs? Some programs will use your default Java, some will invite you to select the JDK to use, some will bundle their own complete JDK. Oh, a program uses JavaFX? No, of course that isn't installed just because you installed Java.

These problems aren't inherent to Java, of course - it's 98% due to Oracle's attempts to inconvenience people into paying for licenses.

Re:Kafka Removing Zookeeper Dependency by elric on Sunday May 17, 2020 @06:37AM (#23211047)
I don't know which JVMs are or aren't in Ubuntu. But Ubuntu 18 is over 2 years old. Almost every major linux distribution has stable, LTS versions of Java 8 and 11 -- which are the currently supported LTS versions in general.

A package can easily add a dependency on a specific Java version. And unlike many other tools, if you have multiple versions of Java, all you usually need to do is add the right one to the $PATH. Sometimes you may have to set $JRE_HOME. And that's it. You're done. If that's too much of a pain to deal with, I hope you never have to install a tool that requires a specific PHP, Python or NodeJS version.

I don't know of any serious tools written in Java that don't either explicitly say which version(s) are supported. Maybe some tools are poorly packaged. But again, that's not a problem with Java.

Re:Kafka Removing Zookeeper Dependency by apta on Sunday May 17, 2020 @11:12AM (#23212536)
You can also ship the JVM directly with your product, that way it's system independent. Even better with jlink.
Re:Kafka Removing Zookeeper Dependency by PopeDotNinja on Sunday May 17, 2020 @06:26AM (#23211007)
+1 for this. A headless install of Java is not straight forward. Doable? Yes. Pain in the butt? Yes.
Re:Kafka Removing Zookeeper Dependency by elric on Sunday May 17, 2020 @06:38AM (#23211056)
yum install java-latest-openjdk-headless

There, you're done. Need fonts? Install fonts. Pain in the butt? Where?

Re:Kafka Removing Zookeeper Dependency by bsaul on Sunday May 17, 2020 @07:27AM (#23211246)
And now you have a separate intermediate runtime layer between your code and the OS, with its own set of versions, patches, regression, bugs, backward compatibilities, etc. that may or may not have an impact on the code that you intend to deploy (or redeploy).

Statically compiled binaries are undoubtedly a plus regarding deployment.

Re:Kafka Removing Zookeeper Dependency by throwdbaaway on Sunday May 17, 2020 @01:21PM (#23213476)
You do realize that golang statically compiled binaries contain the go runtime right? So, all the concerns that you mentioned are applicable to go as well, e.g. a golang app may be buggy when built with version X, while another golang app may have GC issue when built with version Y, and yet another golang app would be vulnerable unless rebuilt with version Z. [1][2][3]

Don't drink the Kool-Aid, golang is just a more opinionated and much less capable cousin of Java/JVM. And for many companies, that could be the right trade-off.

[1] https://groups.google.com/forum/m/#!topic/golang-announce/mV...

[2] https://groups.google.com/forum/m/#!topic/kubernetes-securit...

[3] https://groups.google.com/forum/m/#!topic/golang-announce/65...

Re:Kafka Removing Zookeeper Dependency by bsaul on Sunday May 17, 2020 @02:59PM (#23214338)
except that the developer controls it, since its is embedded. As such it follows the exact same lifecycle as the code itself ( versionning, testing, etc).
Re:Kafka Removing Zookeeper Dependency by vips7L on Sunday May 17, 2020 @11:36PM (#23218059)
This is misinformed. jlink has existed since jdk 11 and allows the developer to produce a custom JVM to ship with their application. jpackage also now exists to create installers for whatever platform your on. It is you the developer who are responsible for packaging your application correctly.
Re:Kafka Removing Zookeeper Dependency by hkeide on Sunday May 24, 2020 @04:55AM (#23290194)
Why doesn't Kafka do this?
Re:Kafka Removing Zookeeper Dependency by ashtonkem on Sunday May 17, 2020 @08:22AM (#23211457)
I’ve deployed Java apps for years, the idea that I must “admit” to Java being painful to deploy in order to be taken seriously is laughable.

Like any other tool, Java has its pros and its cons, but you’re being hyperbolic.

Re:Kafka Removing Zookeeper Dependency by gilbetron on Sunday May 17, 2020 @09:53AM (#23211986)
Ok, you have a java app I want on my system, what are the steps I need to do? (I'm on Ubuntu 18.04)
Re:Kafka Removing Zookeeper Dependency by apta on Sunday May 17, 2020 @11:40AM (#23212709)
You can ship the JVM with your app such that it's self-contained and system independent. Apps like IntelliJ already do this. And now with jlink, you can strip it down to make deployment sizes even smaller by only shipping the parts of the JVM that your app uses.
Re:Kafka Removing Zookeeper Dependency by ashtonkem on Sunday May 17, 2020 @12:33PM (#23213101)
That’s the reasonable choice for consumer facing applications, yeah. For server applications it makes more sense to use something like Docker or a pre-built AMI.

For what I do “how do I get Java on the server” is much less difficult than making deployments quicker and more efficient, which is more a problem for our CI/CD harness and integration tests. Neither of which are Java specific per se.

Re:Kafka Removing Zookeeper Dependency by parasubvert on Monday May 18, 2020 @07:55AM (#23221202)
For server apps you don’t even need a Docker image or AMI, a fat JAR will usually do 90% of what you need.

That said if you want to keep your JDK up to date via Docker or AMI of course that’s fine. a jlink JAR does the same thing but I can see the desire for wanting to decouple runtime upgrades from code upgrades.

Re:Kafka Removing Zookeeper Dependency by splix on Sunday May 17, 2020 @11:40AM (#23212706)
I guess you mean that Java 11 is not available on 18.04 by default, right? Java 11 was release in September 2018, so I guess Ubuntu 18.04, which came in April 2018, didn't have any chance to include it. Therefore you have to make an extra step to setup a ppa repo with software that didn't exist at the moment of release.

Otherwise, at least with Ubuntu 19.04 which I have, it's simple `sudo apt install openjdk-11-jdk-headless`

Re:Kafka Removing Zookeeper Dependency by ashtonkem on Sunday May 17, 2020 @12:26PM (#23213022)
wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/publi... | sudo apt-key add -

sudo add-apt-repository --yes https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/

sudo apt-get install adoptopenjdk-<version>-hotspot

Yes, it’s a few extra steps, but it’s not exactly Sisyphean now is it? Given the complexity of modern CI/CD pipelines, and the trend towards continuous deployment, making sure that a JVM is deployed alongside the uberjar is a pretty small ask. Would a single binary be easier? Sure. Would I change languages just for that? No, other factors are more important IMHO.

Re:Kafka Removing Zookeeper Dependency by karmakaze on Sunday May 17, 2020 @02:39AM (#23210074)
To run a Java jar-with-dependencies you only need to `java -jar app-with-deps.jar` which isn't really that hard at all.
Re:Kafka Removing Zookeeper Dependency by axaxs on Sunday May 17, 2020 @02:54AM (#23210148)
This skips the steps of choosing and installing a JVM, which is a bit daunting. My distro gives like 4 different versions. And worse, some applications only run on certain versions. By default, there is no 'java' command.
Re:Kafka Removing Zookeeper Dependency by weego on Sunday May 17, 2020 @03:39AM (#23210386)
That's not really a good faith argument. That's the equivalent of saying that a major problem with NodeJS is you have to pick which version to install before you deploy.

I deploy single package jvm files most days having not had to consider the deploy environment for a about 4 years.

Re:Kafka Removing Zookeeper Dependency by jooz on Sunday May 17, 2020 @04:45AM (#23210636)
What is hard is not to choose the java version (1.8,19,...), but the JVM implementation. To choose between hotspot, openjdk, oracle. Even to find the download link without being forced to log in on oracle.com was difficult (in fact I couldnt find it the very last time).
Re:Kafka Removing Zookeeper Dependency by npstr on Sunday May 17, 2020 @06:21AM (#23210992)
Have a look at https://sdkman.io
Re:Kafka Removing Zookeeper Dependency by Groxx on Sunday May 17, 2020 @04:05PM (#23214926)
Why is it hard? Since Go gives you no choice, the equivalent would be to pick literally any, and make do with its defaults. Same thing for tuning. You do not have to choose optimally, that's an extra constraint.
Re:Kafka Removing Zookeeper Dependency by femiagbabiaka on Sunday May 17, 2020 @04:42AM (#23210619)
It is a good faith argument — it’s literally the whole point. Native binaries bypass all of that.
Re:Kafka Removing Zookeeper Dependency by Tsarbomb on Sunday May 17, 2020 @09:22AM (#23211791)
Good luck when you need to deploy to a system with a different version of libc.
Re:Kafka Removing Zookeeper Dependency by splix on Sunday May 17, 2020 @11:41AM (#23212716)
Also with different version of openssl
Re:Kafka Removing Zookeeper Dependency by femiagbabiaka on Sunday May 17, 2020 @02:19PM (#23213970)
goalposts are moving at a rapid pace
Re:Kafka Removing Zookeeper Dependency by tamale on Monday May 18, 2020 @09:51PM (#23230523)
having a different version of libc is exactly like worrying that the system has a different version of java. If you know about these 'gotchas', they're not big deal. If you don't, then ya, they can be a bit intimidating at first.

I really don't think running a go app vs. a java app is ANY different.

Re:Kafka Removing Zookeeper Dependency by femiagbabiaka on Tuesday May 19, 2020 @03:50AM (#23232373)
I’m trying to figure out what environments you’re working in where libc incompatibilities are as common as java versioning issues

You’ve made a straw man and are now working very hard to defend it.

Re:Kafka Removing Zookeeper Dependency by morelisp on Sunday May 17, 2020 @06:12AM (#23210966)
> That's the equivalent of saying that a major problem with NodeJS is you have to pick which version to install before you deploy.

Are you saying this isn't a major problem with Node? In my experience it's a problem with every language that requires a separate runtime.

The JVM is the worst of two worlds because I have the build complexity of an AOT compiler to make my deployment artifact and the deployment complexity of an interpreted language to get the correct runtime pre-installed.

Re:Kafka Removing Zookeeper Dependency by Tsarbomb on Sunday May 17, 2020 @09:26AM (#23211816)
You do realize Java has maintained it's backwards compatibility making choosing your deployment runtime significantly easier.
Re:Kafka Removing Zookeeper Dependency by karmakaze on Sunday May 17, 2020 @11:23AM (#23212595)
This is a question you have to answer once for your project. Similar to choosing which OS, database, library, or framework to use. There's no avoiding having knowledge or learning when choosing dependencies.

Just about any of the implentations will work fine. You only have to concern yourself with avoiding use of the Oracle runtime in production without a license. If in doubt, install the latest AdoptOpenJDK[0] Hotspot version that's compatible with your app.

[0] https://adoptopenjdk.net/

Re:Kafka Removing Zookeeper Dependency by apta on Sunday May 17, 2020 @11:42AM (#23212719)
You can ship the JVM with your app such that it's self-contained and system independent. Apps like IntelliJ already do this. And now with jlink, you can strip it down to make deployment sizes even smaller by only shipping the parts of the JVM that your app uses.
Re:Kafka Removing Zookeeper Dependency by devonkim on Sunday May 17, 2020 @10:10AM (#23212108)
It depends upon what you’re looking at in a deployment. Static binaries are the gold standard of “simple” in theory for ops folks but are not architecture neutral by definition. The JVM JAR spec is pretty well known as well and is not a surprise with plenty of documentation or blog posts to help out. Quirks do show up from crossing boundaries into native land like DNS and sockets behaviors, but a basic JAR does give some advantages over a random native binary in a tarball when it comes to dependency management which is tougher with a static binary. It’s not obvious which version of a library is compiled into a static binary at first glance compared to unzipping and checksumming the JARs in a package.

There’s definitely some laborious parts of the Java packaging setup (the whole resources, meta-inf / manifest setup reeks of YAGNI problems) but similar to Go’s lack of expressiveness being a feature, the Java ecosystem is fundamentally designed for organizations that separate developers from the systems where the software is run, and this is either helpful or hurtful entirely depending upon the organization’s needs.

I’ve never really had a problem deploying anything based upon the packaging - it’s a minor part compared to various obscure configuration files, ConfigMaps, or environment variable injections that bother me more, and that has nothing to do with a package or even language in itself.

Re:Kafka Removing Zookeeper Dependency by skywhopper on Sunday May 17, 2020 @11:18AM (#23212560)
Wow, this does not match my experience at all. The JVM itself is harder to install than any native-binary service I've ever used. And that's before you get around to running something like Kafka or Zookeeper on top of it. Maybe "not all things" are a pain to deploy, but everything I've ever run on it or looked into running has been.

I'm particularly curious about how you deal with TLS in the JVM without keystores? I'd love to hear more about that.

Also, I have to laugh about the heap settings not "technically" being required. Sure, but how many JVM-based services have you actually run in production without setting them? If it's more than zero, then I congratulate you on your luck.

Re:Kafka Removing Zookeeper Dependency by elric on Monday May 18, 2020 @01:40AM (#23218762)
How are you installing these troublesome JVMs? Either you use your OS' package manager and get updates as a bonus (for a looooong time on CentOS/RHEL). Or you can literally download an OpenJDK zipfile, unzip it and you're done -- will require manual updating. Or, if you're feeling enterpricey (not a typo), you can install an Oracle JVM and get paid updates for an eternity.

Kafka and ZK are a pain to deploy. Not the JVM's fault. Entirely down to the choices made by their respective community. Last I checked it was still virtually impossible to secure a ZK-ensemble. Maybe they'd welcome patches, but when I tried to submit a minor bugfix years ago, I found them to be an unwelcoming community.