The reason packaging is still a problem in Python are linked to the language:
- to get perfs, you need compiled extensions, which java doesn't need. So a Jar can be fast and portable, while in Python it's an "or" proposition. Wheels help, but they still are OS dependent, so you need a build per OS.
- PYZ arrived very late in Python, while WAR were a very early thing in Java. Few people even know you can distribute a whole Python program in a single zip file with dependencies using something like shiv (shiv.readthedocs.io). So few people do it, and it's a shame. We do it for 0bin though: https://github.com/Tygs/0bin/releases. And it doesn't solve the wheel problem, although nuitka does but it a higher prices to pay.
- packaging tooling have been a mess for 2 decades. Setuptools/distutils/setuptools2, easy_install/pip, and setup.cfg/setup.py/pyproject.toml. We failed as a community on this one. It's getting better, but it's a slow process.
- the Python community is at least half composed of amateurs. It's a good thing for the dynamism of the community, but it also means the software are not very polished. Hard packaging + amateurs = packages that are releases when they reach "just good enough" stage, not one inch more. It also means you'll get things like leaking stack traces, config files in src and the likes.
So yeah, compared to Java, Python packaging and distribution sucks.
But compared to rust all packaging and distribution for all languages do, so there is that.
or, even worse, scientists.
The previous crowd with love for monkeypatching everything and hate for abstraction other than dictionaries wasn't too pleasant, too.
Code style! Hah. First day on the job I asked about that and was told nobody cared and everybody had their own preferences.
... I'm very bitter today. I'm working on it.
Everything is jupyter notebooks, to the point where example documentation is delivered as notebooks and services run notebooks as it’s apparently easier to do that than to get people to write their code properly. “Oh but data exploration” they say, which is fine, but you shouldn’t be doing data exploration in your prod ML pipeline anyway, and you’re writing code, it’s really not that hard to follow some even rudimentary practices.
I'm forced to use Python at work, because data scientists demand it, and I can rant about your "good for data" language for at least 24 hours straight.
Python isn't great at anything specifically, but it's crap in a few specific places. No multithreading? Fine! But then Python core libraries are crap at IPC, when multiprocess is the preferred way of parallel computing!
I don't know where you read either of your propositions in any of my comment.
You've listed things that rely on full stack of libraries and deployment options.(except scripting) As someone who has the pleasure to work with Python - Python is horrible at a lot of these things, in spite of the beauty of the language itself.
Python is the best language in the world for small tools under 500 lines to solve glue and data munging problems.
There are two main exceptions to this rule. One is when an actual software engineer ends up working as a scientist and having to write papers (like me). Then, the paper is more likely to be about the software itself, and the citations of the paper are from people who have also used the same software. This creates an incentive to make the software high quality, in order to get those citations. The other exception is when an institute decides that the world needs a decent piece of software to fulfil a vital role, and actually funds its development.
Oh, I should probably mention - I just helped out with a programming course for undergraduate medical science students. Most of the time I was tearing my hair out about the terrible programming practices being taught by the module leader, who admittedly had absolutely no computer science qualifications whatsoever (and in fact didn't know the difference between a forward slash and a backslash).
I think a solution to this is to recognise software quality as a common benefit, and divert funding towards it, allowing real software engineers to have an actual meaningful career in university. Universities are usually all over common benefit stuff, so you would have thought this would be easy enough. The Research Software Engineer programme in the UK is a start.
Pyinstaller builds bundle the python binary with the project. It works without a Python installed on the system, is not portable and the resulting binary is fat. It also can fail in numerous way and be slow, but end users will find it the easier to use when they work.
Pyz files just contains python code. Shiv produce them by bundling your code and the dependencies, but you still need a Python installed on the system to run that. You pay the cost of zipping only on the first run though, after that, it's unzipped. And their are portable if you use no binary wheel. It's light and easy to get right.
In that sense, they look more like War files in Java.
Pyinstaller, like cx_freeze and nuikta are for end users.
Pyz are for devs.
> with Java, you just dump the necessary jars into a directory, include them in the classpath, and it works.
Ant, maven, or gradle, and then some jars, somewhere? What's the classpath and whats $JAVA_HOME for this project?
Oh and now I have to make it work in the IDE too, click through a billion menus and dialogs, sync the build system, sync the dependencies, configure runtimes and classpaths, try to rebuild for the 10th time, pray.
> Oh and now I have to make it work in the IDE too, click through a billion menus and dialogs,
You click "Project from existing sources" in IntellIJ and it will recognize both Maven and Gradle projects and just work?
Sync the dependencies? configure runtimes and classpaths? What? With Maven and Gradle? Your talking Ant here.
- Install and set the right java version system-wide
- Copy some config files for the project system-wide (granted, this is not java specific, but still)
- Click in some IDE menu to enable some build settings
- Click in some IDE menu to install some "Lombok" plugin
- Click in some IDE menu to select the right jdk
- Click in some IDE menu to edit the run configuration:
- select main class
- paste program args
- paste env vars
- select right “Use classpath of module:” option
- Click in some IDE menu to fiddle with "Spring Loader path": modules, dependencies, reorder the dependencies (!)
- Click in some IDE menu to configure getting the actual logs
- Click maven-sync, rebuild-project multiple times, restart IDE until it suddenly works
I know you will say, this is all fault of whoever made this crap project, or my fault for not fixing it. However this was my experience, and it has been similar every time in my life that I had to touch some java project (although this last one is the worst).
In my experience (sample size of hundreds of projects), Maven projects are <<by far>>, on average, easier to build and manage than most other languages. The repo structure is standardized, the build cycle is standardized, the built packages are always in the same place, etc.
The Java installation is a 1-time think. It's also a quick google away. I see this complaint a lot, I never understand it. It's plain and simple bikeshedding: someone complaining about a 1-time task that takes 5 minutes in total.
The rest of the stuff is IDE specific and I have no idea why you'd have to configure "getting the actual logs"?
Treating the IDE like a separate problem is not realistic with java.
With e.g. Rust, I can use vim, vscode, whatever floats your boat.
> no idea why you'd have to configure "getting the actual logs"?
Some genius java logging library, I can only guess.
With Rust you code in VIM because there's nothing better. Improving Rust IDE's has been one of the teams big focuses for years. Everyone codes in random text editors because there's no amazing IDE.
You don't need to constantly "extract method", move to class, generate all these mountains of classes by rightclick. This says a lot about the language.
But the main point is that with rust it actually just works: there is one way to set global toolchain and per project/dir. You're not required to also click through a million menus and dialogs and copypaste stuff and sync and rebuild and restart to get the editor barely working.
In Rust, macros generate code on build. In Java, Annotations Processors do. I don't see a big difference between these approaches, besides Java's mechanism being older and crustier. Rust macros generate tons of code.
> if you have rust-analyze you get inline errors, code completion, jump to definition, autoimport, autoformat...
Last time I checked, Rust analyzer was in alpha/beta. Java has all this and it's battle-tested. Including language server for VScode and other lightweight editors, and several options for build plugins that auto format.
> You don't need to constantly "extract method", move to class, generate all these mountains of classes by rightclick.
Lombok, or one of the popular Builder libraries, gets rid of all this boilerplate. I don't generate anything using IDE. In my experience, most Java user don't. There's many mature libraries for geting around class ceremony.
> there is one way to set global toolchain and per project/dir. You're not required to also click through a million menus and dialogs and copypaste stuff and sync and rebuild and restart to get the editor barely working.
You are right. Rusts build system is simpler and better integrated. But Java has amazing build tools, maybe the best out there. A properly configured maven project will compile, test, and run with one command. And it will import into your IDE with a simple "import project, yes"
I have worked on a ton of Java projects over the years and never had to deal with most of what your describe.
True, but with IntelliJ 99% of what you wrote is auto-configured.
Eclipse needs a bit of hand-holding, but almost everything you described is due to project configuration.
> Some genius java logging library, I can only guess.
That's your problem, you're assuming that Java is dumb and its libraries are dumb. They're not. I'm reasonably sure that <<all>> mainstream Java logging libraries log to console by default.
The previous engineer must have been a marvel of a developer to manage to screw even that up.
What I'd say is: Java, especially older enterprise code, had a ton of over-engineered patterns. These are slowly fading away, except for Spring, and even Spring's simpler these days. Java also tends to have some low quality and a ton of average but unmotivated working devs on Java projects. So yeah, there's that ecosystem problem.
But for example, if you'd start a new project and you approach Java with an open mind, it's one of the most powerful and flexible techs available. And it's much more modern than the stereotypes imply.
It is not, that's what I'm saying. And I don't understand why, if IntelliJ is the de-facto IDE. I'm sure that a fresh project is a one-click import, but an average project that isn't fresh anymore, always have similar IDE-setup-magic problems. At least in my experience over the years.
I know plenty of people who don't use a formal IDE. This "I don't use an IDE" is a stupid flex.
It's not a flex, simply a matter of preference: I find using the mouse too irritating, others prefer it because it's simpler.
But that's not my gripe, my gripe is that setting up IntelliJ, the one defacto standard IDE to actually work is some voodoo ritual of a dozen steps and restarts and so on.
However I'd expect Lombok users to outnumber Lombok devs 100k to 1 :-)
I frequently find that the particular constellation of build tooling and libraries particular projects have requires a particular version of the jvm and all the environment variables set up correctly. In some other languages there is widely used tooling to manage the install and use of the appropriate platform version for the project.
If you're working with a variety of projects, fiddling with the installed jvm and java_home and friends is not a one time thing.
It is. There's really no excuse for that. If it's been similar every time then you've worked on crap every time.
Building a java project should be (and is on projects I work on) as easy as installing a jvm and running either maven or gradle to build it. End of story.
"Install some lombok plugin" is only necessary if you want the IDE to understand lombok, IntelliJ will just pop up a box saying "Want this?" and you click "Yes". I don't see what's so onerous about that.
The rest you describe is really down to poor developers.
So "no offense", but shove it.
Yeah that’s a crap project. My experience was always that mvn install / ./gradlew run will just work. On all three OSs. And if I happen upon a decade old random jar file, and run java -jar something.jar, it will just work in the great majority of time. I think no other platform has it this smooth.
The project could have been set up instead with the "Maven wrapper"(1) which provides "mvn.cmd/sh" scripts at the top level of the project to fetch and run the proper Maven. So after cloning the project, all you have to do is set a proper JAVA_HOME if it's not already set, and then do "./mvn clean package" at the top of the project to do a complete build. The IDE's will then set themselves up properly if you import the project as a Maven project, and will sync with changes to the Maven pom.xml whenever it changes (Intellij does at least). That's really hard to beat.
(1) https://github.com/takari/maven-wrapper (Being integrated into Maven itself soon).
We occasionally do remote debugging at work. It’s just a matter of copy/pasting the correct JVM incantation into the IDE.
Even if your project is so messy, then you should be able to just to version a working setup (maybe needing to figure out how to make the correct JDK version be used across different machines) and after that it'd become a question of just having to check out the project from the version control system and click a green button in IntelliJ.
This most likely wouldn't solve all of your issues, but at least would help make onboarding slightly less painful and slow.
The run configuration can be automatically created by Intellij by attempting to execute a runnable class. You only need to set program args and env vars if needed - and that would be true for ANY language or program.
The spring loader path appears to be specific for this project.
The run logs are independent of Java. Thats just getting program logs.
There is no need to maven-sync and rebuild-project multiple times, unless you are explicitly refactoring the project module/build structure. Once you have imported a maven project, you can simply click build.
Create new Java project in IDE Add no dependencies Use the built in IDE tooling to create a runnable JAR and tick the box to export the Ant build to a build.xml file.
Why would I want to complicate things by adding Maven/Gradle to the mix?
And I will also give you the secret trick of superninja senior developpers: be the one who manages the build process, so you will never EVER be fired.
Non-networked applications don't tend to need to be so paranoid about keeping up to date with the latest versions of dependencies.
My point is there are a wide variety of application types out there, not everything is a web application, not everything needs to use a complex build and dependency management system.
So just use Maven and save anyone coming after you a bunch of trouble when they have to extend your project ;-)
Dependency management is a mess in the Python world and my number 1 complaint. You have pip, pipenv, poetry, anaconda, to manage your 3rd parties. And if you want to avoid conflicts and have consistent runtime you need to create a virtual environment to install these libraries and redownload everything.
Otherwise libraries are installed user or system wide, in a directory that depends on the Python version that you use but they are not versioned there.
With Maven you don´t have the issue. Your 3rd parties are downloaded in a global cache directory (the Maven repo) and are all versioned. For the delivery you can either provide them all and have the runner set the CLASSPATH or have them packaged into a single jar.
I am not saying Python's packaging is any good, it sucks. But you can't realistically ignore Java's IDE shitshow.
All needed is having the correct version of Java, and maybe give maven a password so it can access private libraries. It's dead simple.
And more importantly: it's the same everywhere.
I haven't used Maven in a bunch of years.
But source code goes in src/main/java. Test code goes in src/test/java, resources (images, files, whatever) go in src/main/resources, test resources go in src/test/resources. There is a pom.xml file that contains dependencies, and if it doesn't, it points to a parent which <<does>> contain them.
There might be submodules and they all are folders containing another pom.xml. That pom.xml generally points to the main pom.xml.
Etc. There's a bit of tedium in there, but <<it's always the same>>!
The level of consistency is something most ecosystems dream about.
And you know what? That means that you can use your spare brain cells for actual creativity or hard work.
That is not my experience at all. I dont have to rebuild work project many times. Creating new scratch project for playing in is easy and fast. Dependencies are managed by maven and I rarely have to deal with that beyond added dependency tag.
Maven can build a nice fat executable jar, with just one XML block added to the pom.
I've opened a 5year unmaintained project with IntelliJ yesterday - click package and got a working executable.
I opened another project built with SBT... and guess what? No errors!
Let's try that with a Python project (LOL!)
That's an interesting take; my experience has been (at least the community) is quite proudly dogmatic on the One Right Way To Do It, for all values of "It".
I also don't like Python, but I think for different reasons and perhaps a bit less than you.
Never used R, but I understand it to be quite fiddly.
This hasn't been true lately. Python is growing ever more complex.
But, for people still working in "computer science-y" domains, I haven't seen the historical dogmatism lessen; in many cases people are digging in.
In other words R is wrong tool to use for building the software just from maintainability standpoint.
I've seen very complex R packages that have been around for 30+ years. They are actively maintained. Applying good design principles, unit testing, modularity - all that is very much doable in R.
And you will want to write software in R, especially if the purpose is to support the very activity of working with R packages and to expose the R statistical environment over HTTP. Case in point, the software I link to, in my comment above.
As a Java/Python/Go developer I can say that packaging is a solved problem as of 2021.
I like to point out Airflow as an example. It's still a complete nightmare to deal with and it has many steps for just packaging the application.
https://github.com/apache/airflow
I have less trouble packaging and running a 3 year unmaintained Maven Java application, than actively maintained Airflow. And let's not forget that I can target Java 8, without having to install Java 8 on my machine. (Which only adds to Python's complexity)
Like you mentioned, wheel files are not nearly as customizable as JARs. You can easily pick & choose the dependencies to include in JARs (fat JARs) and change namespaces of dependencies via shading to prevent collisions.
I've generally found it easier to add JARs to Spark clusters compared to attaching wheel files to PySpark clusters.
It's my guess that you are using a lot of Python libraries that are wrappers around C or C++ libraries which have dodgy cross-platform support, because I can't see another way this is happening. If so, it's hardly fair to blame Python for that...
Java is far worse on this front. Let's take a look at JEE or Spring - The entire experience is akin to summoning a demon with waving reciting correct incantations in correct places while having no idea about what even goes on in the background. There are MessageChannels, MessageProducers, ClientFactories that interact in mysterious ways and initiated in also very mysterious ways. And then there's EJB's and their containers which are just blackboxes of automagic.
Or how about Hibernate?
It now has 2 dependencies. It is a 1000 times easier to use than hibernate, despite being one of the worse ORM in Python.
That the problem with old Java libs like Swing or Hibernate: they have been designed for the 90' and don't leverage the potential of what a modern Java version can do.
There are many modern java libraries that are very lean and mean out there and package into small-sized redistributable assemblies.
But my main complaint about Python is trying to use someone else's software. Then it degenerates into a maze of twisty little passages of trying to install the correct set of dependencies, not knowing where those files are installed (which makes it difficult on a cluster, as you may need to do the install on every node separately). Then you try using pip and it fails because some element of what it is trying to install isn't compatible with your operating system for reason 1. So you try anaconda instead, and that fails in a different way, like it runs a C compiler on some file (!?) which fails because $REASON. I have numerous examples of Python software where I have followed the (detailed, precise) installation instructions, and it doesn't work.
Whereas with Java, you just dump the necessary jars into a directory, include them in the classpath, and it works.
Don't get me started on R.