Re:Java Is Underhyped
by MrPowers on Friday April 16, 2021 @06:17AM
(#26831975)
Python's packaging is a lot better with Poetry. Use Poetry and build the wheel with poetry build. The lockfile lets you create an exact virtual environment.
Like you mentioned, wheel files are not nearly as customizable as JARs. You can easily pick & choose the dependencies to include in JARs (fat JARs) and change namespaces of dependencies via shading to prevent collisions.
I've generally found it easier to add JARs to Spark clusters compared to attaching wheel files to PySpark clusters.
Re:Java Is Underhyped
by dunefox on Friday April 16, 2021 @08:51AM
(#26833246)
I'm all for a good project-based package manager for Python, but I have tried Poetry several times over the last years and not once has it worked as intended. It always failed to install packages for some reason.
Like you mentioned, wheel files are not nearly as customizable as JARs. You can easily pick & choose the dependencies to include in JARs (fat JARs) and change namespaces of dependencies via shading to prevent collisions.
I've generally found it easier to add JARs to Spark clusters compared to attaching wheel files to PySpark clusters.