What you described are multiple tools that also target different areas:
> - poetry
from what you listed this seems like the only tool that actually takes care of dependency management
> - "Just pin the dependencies and use Docker"
this is standard fallback for all languages when people are lazy and don't want to figure out how to handle the dependencies
> - pip freeze
all this does it just lists currently installed packages in a form that can be automatically read by pip
> - Vendoring in dependency code
this again is just a way that applies to all languages, and it is still necessary even if there's a robust dependency management as there are some cases where bundling everything together is preferred
> - pipreqs
this is just a tool that scans your code and tells you what dependencies you are using. You are really lost if you need a tool to tell you what packages is your application is using, but I suppose it can be useful for one offs if you inherit some python code that wasn't using any dependence management.
> - virtualenv
this is just a method to have dependencies installed locally in project directory instead per system. This was created especially for development (although it can be used for deployment as well) as people started working on multiple services with different dependencies. It's now included in python so it's more like a feature of the language.
Package management is pretty much a solved problem, no matter how old is your language. It smells to an outsider like me like a lot of bike-shedding and not enough pragmatism is going on in Python land over this issue. Has a new BDFL stepped up after Guido left?
I think also a lot of issues with packaging is ironically because of PyPA that supposed to work on a standard, but in reality instead of embracing and promoting something that works they just pushes half-assed solutions because author is one of the members. Kind of like they were pushing failed Pipenv "for humans". Seems like Poetry is generally popular and devs are happy with it, so of course PyPA started pushing their own Hatch project, because python packaging was finally getting too straight forward.
I think Python would benefit as a whole if PyPA was just dissolved.
Reading the comment thread it's not immediately clear what the answer is - it seems implied that the proper way is using Poetry, is that the case?
See https://imgs.xkcd.com/comics/python_environment_2x.png
Or sometime the computer is haunted and my colleague had problems installing tensorflow. To this day he has no working tensorflow.
Please note I never used tensorflow on that computer in fact I never used tensorflow, but this is my experience: https://gist.github.com/takeda/89ec29501b6e8641415668f22f3e9...
It succeeded after first try.
I do see that in their repo[1] they use a non standard way to build the package. They use Bazel, but that's Google for you. They never do things everyone else is doing. I'm not sure why this is Python problem rather than package problem.
They have tons of open issues around building: [2]
[1] https://github.com/tensorflow/tensorflow
[2] https://github.com/tensorflow/tensorflow/labels/type%3Abuild...
It works fine for me.
Not for him. Same package, nearly identical laptops, different outcomes.
And I as newcomer to Python, there are like four (pip, venv, brew, conda) cli APIs that I had to learn just to get working on some Python file.
STRONG agree. And I hate the cult of personality that was previously (still?) strong.
Not sure anyone was asking for that, unlike a fix for packaging issues.
And worse, effects code maintainability - if you need that assignment higher up, you're now editing the if statement, adding an assignment, plus whatever your interstitial code is.
Python doesn't have block scoping so the argument for it is weak.
> And worse, effects code maintainability - if you need that assignment higher up, you're now editing the if statement, adding an assignment, plus whatever your interstitial code is.
How is that different than variables declared without the walrus operator? If you declare a variable with the walrus operator and decide to move its declaration you can still continue to reference that variable in the same spot, just like any other variable. Do you have an example you can share to demonstrate this? I'm not sure I understand what you mean.
> Python doesn't have block scoping so the argument for it is weak.
The walrus operator another way to define variables, not change how they behave. It's just another addition to the "pythonic" way of coding. It's helped me to write more concise and even clearer code. I suggest reading the Effective Python link I provided for some examples of how you can benefit from it.
# some other code
if determined_value := some_function_call():
do_action(determined_value)
and then I change it to this: # some other code
determined_value = some_function_call()
logger.info("Determined value was %s", determined_value)
validate(determined_value)
if determined_value:
do_action(determined_value)
and determined_value is a reasonably expensive operation (at the very least I would never want to redundantly do it twice) - then in this case my diff for this looks like: --- <unnamed>
+++ <unnamed>
@@ -1,5 +1,8 @@
-
# some other code
-if determined_value := some_function_call():
+determined_value = some_function_call()
+logger.info("Determined value was %s", determined_value)
+validate(determined_value)
+
+if determined_value:
do_action(determined_value)
whereas if I wrote it without walrus originally: --- <unnamed>
+++ <unnamed>
@@ -1,5 +1,8 @@
# some other code
determined_value = some_function_call()
+logger.info("Determined value was %s", determined_value)
+validate(determined_value)
+
if determined_value:
do_action(determined_value)
then the diff is easier to read, and the intent is clearer because diff can simply infer that what's happening is the semantic addition of two lines.Code is read more then it's written, and changed more then originally created, and making the change case clearer makes sense.
This is such a pedantic, non-issue I don't even know why I'm bothering acknowledging it.
while (n := s.read(buffer)) != 0:
#do something with first n bytes of buffer
Without the walrus operator you either have to duplicate the read operation before the loop and at the end of the loop, or use while True with a break if n is zero. Both of which are ugly IMO.> all this does it just lists currently installed packages in a form that can be automatically read by pip
Are you referring to version specifiers[1] being optional or is there something more to versions that I don't understand? PEP 440 is a wall of text, maybe I should get around to reading it sometime.
[1] https://packaging.python.org/en/latest/glossary/#term-Versio...
I tried to say that from mentioned tools only poetry can be called as a dependency management.
The other tools are used for different purposes, but perhaps could be used as a piece of package management in some way. The mentioned docker and vendoring is irrelevant to Python and it even applies to Go.
Granted, you don't get a nice executable, but it's still miles ahead of C++ (people literally put their code into header files so you don't have to link to a library), and even modern languages like rust (stuff is always broken, or I have some incompatible version, even when it builds it doesn't work)
By the way if you're a Python user, Nim is worth checking out. It's compiled, fast and very low fuss kind of language that looks a lot like Python.
Been working on and off with Rust for the last 3 years, never happened to me once -- with the exception of the Tokio async runtime that has breaking changes between versions. Everything else always worked on the first try (checked with tests, too).
Comparing Python with C++ doesn't make do argument any favours, and neither does stretching a single Rust accident to mean the ecosystem is bad.
For crates that don't follow semver (which I'm fairly certain I've encountered zero times) you can pin a specific exact version.
I will say, though, that this only accounts for times where you’re not upgrading dependencies. Where I’ve always run into issues in Python was when I decided to upgrade a dependency and eventually trigger some impossible mess.
In Elixir I can do `mix hex.outdated` in any project, no matter who wrote it, and it'll tell me very quickly what's safe to update and link to a diff of all the code changes. It's night and day.
Thankfully, it's getting gradually better with poetry, but it's still quite clunky compared to what you get elsewhere. I noticed lately for instance that the silent flag is broken, and there's apparently no way to prevent it from spamming 10k lines of progress bars in the CI logs. There's an issue on Github, lost in a sea of 916 other open issues...
If A and B both depend on different versions of C, then both versions of C are installed and A/B see the version they want.
I can't see a reason why Go would be different.
I have rarely encountered issues in rust. Most rust crates stick to semver so you know when there will be a breaking change. My rust experience with Cargo can only be described as problem free(though I only develop for x86 linux).
As for pip freeze and virtualenv things start to fall apart especially quickly when you require various C/C++ dependencies (which in various parts of the ecosystem is a lot) as well as different python versions (if you are supporting any kind of legacy software). This is also assuming other people working on the same project have the same python yadda yadda the list goes on, its not great.
Yes 100 times. That can be incredibly frustrating. During the last year I've used a large (and rapidly evolving) C++ project (with many foss C/C++ dependencies) with Python bindings. We've collectively wasted sooo many hours on dependency issues in the team.
Long compilation times contribute considerably to slow down the feedback loop when debugging the issues.
It's a very good argument.
- Pip freeze just pins all dependencies at once to requirements.txt
- I don't know what "vendoring in dependency code" means
- I've never used pipreqs in my life (and 80% of my work has been in Python)
- Virtualenvs are just a convenient way to keep project runtimes separated
And for 90% of Python projects in existence the following is sufficient (assuming Python3 is installed):
- python -m venv .venv
- source .venv/bin/activate
- pip install -r requirements.txt
That's it. And all of that requires a single dependency: Python. Could it be better? Sure. But to call that a "mess" is an exaggeration.
>- source .venv/bin/activate
If those two steps were automatic like in node.js we'd literally have 99% less problems.
pyenv install 3.9.47
pyenv virtualenv 3.9.47 my app
git clone …/myapp
cd …/myapp
pyenv local myapp
pip install -r requirements.txt
Is it annoying, maybe, but I normally don’t trust system deps for anything. Solving environment |
Solving environment /
Solving environment -
Solving environment \
Solving environment |
...
One day it takes 10 seconds, next month it takes 10 minutes, and the month after that it takes 30 minutes and then fails entirely.Since you specifically mentioned 2021 instead of 2022, I half suspect you had the same experience.
To answer your question: yes, we were using conda-forge, and then when it stopped building we moved to a mix of conda and a critical subchannel, and then a major GIS library broke on conda and stayed that way for months so we threw in the towel and just used pip + a few touch-up scripts. Now that everyone else has followed suit, pip is the place where things "just work" so now we just use pip, no touchups required.
Of course the answer to these questions could be anything but to me it feels like attacks on Python's package management are usually cheap shots on a much much more complicated problem domain than the "complainers" are typically aware of.
¯\_(ツ)_/¯
Your comment hints that you are feeling personally attacked when Python is criticized. Friendly unsolicited advice: don't do that, it's not healthy for you.
Python is a relic. Its popularity and integration with super-strong C/C++ libraries has been carrying it for at least the last 5 years, if not 10. There's no mystery: it's a network effect. Quality is irrelevant when something is popular.
And yes I used Python. Hated it every time. I guess I have to thank Python for learning bash scripting well. I still ended up wasting less time.
Use Python if it's useful for you, obviously. To me though the writing is on the wall -- it's on its loooong and painful (due to people being in denial) way out.
EDIT: I don't "hate"; it was a figure of speech. Our work has no place for such emotions. I simply get "sick of" (read: become weary of) something being preached as good when it clearly is not, at least in purely technical terms. And the "hate" is not at all unfounded. Choosing to ignore what doesn't conform to your view is not an argument.
I am not feeling personally attacked (I am not married to Python), I am mostly just tired of reading the same unproductive type of complaints over and over again. This attitude is not unique to Python's situation, but actually is typical to our industry. It makes me want to find a different job, on some days.
The community is trying to improve the situation but there is no way to erase Python's history. So it's always going to continue to look messy if you keep looking back. The complaint is unproductive, or in other words, not constructive.
You might not be married to Python but it sure looks that way for many others. I switched main languages no less than 4 times in my career and each time it was an objective improvement.
The thing that kind of makes me look down on other programmers is them refusing to move on.
I still dabble in all of them. Who knows when I will move on to the next. Rust looks nice. I tried go.
But they do not yet provide any of the tools/libraries I need for my work. That's how I've always selected my programming language.
So I would first need to invent the universe before I can create valuable things. Instead I will just wait until their ecosystems mature a little more.
I will end the discussion here though. Thanks for the response!
As a rule elixir devs don't do system level dependencies, probably because of lessons learned from the hell scape that is python (and Ruby)
Yesterday I onboarded a coworker onto the elixir project, he instinctively put it into a docker container. I laughed and just told him to run it bare (he runs macos, I run Linux). There were 0 problems out of the box except I forgot the npm incantations to load up the frontend libraries.
- Can you resolve precompiled GPU dependencies with system managed CUDA driver versions?
- Are there packages that can convert PDF to PNG without system dependencies?
- Can you run a Qt GUI app on CI without needing to do any additional system setup?
With regards to cross compilation tools like burrito, that's neat! But Python is not a compiled language.
2) not that I can find for that specific task but the typical strategy is to download (or compile) a binary, drop it into a {project-dependency}[0]-local "private assets" directory, and call out the binary. This is for example how I embed zig pl into elixir (see "zigler") without system-level dependencies. Setting this up is about 40 lines of code.
3) wx is preferred in the ecosystem over qt, but this (and openssl) are the two biggies in terms of "needs system deps", though it's possible to run without wx.
For native graphics, elixir is treading towards glfw, which doesn't have widgets, but from what I hear there are very few if any gotchas in terms of using it.
I bring up cross-compilation, because burrito allows you to cross-compile natively implemented code, e.g. bcrypt that's in a library. So libraries that need c-FFI typically don't ship binaries, they compile at build time. Burrito binds "the correct" architecture into the c-flags and enables you to cross compile c-FFI stuff, so you don't have a system level dependency.
[0] not that this has happened, but two dependencies in the same project could download different versions of the same binary and not collide with each other.
Coming to that from hearing stories that there was supposed to be one way to do everything disenchanted me quickly.
The code itself was okay, but everything around it was a train wreck compared to every other language I’d been using (Go, Java, Ruby, Elixir, even Perl).
I attempted to get into it based on good things I’d heard online, but in the end it just wasn’t my cup of tea.
I don't agree.
The problem is simply that Python encompasses a MUCH larger space with "package management" than most languages. It also has been around long enough to generate fairly deep dependency chains.
As a counterexample, try using rust-analyzer or rust-skia on a Beaglebone Black. Good luck. Whereas, my Python stuff runs flawlessly.
What many newer languages do is precompile the happy paths(x86, x86-64, and arm64) and then hang you out to dry if that's not what you are on.
After an few hours or so of not making much progress in AOC day 1 I just gave up and never continued learning Go.
These are two different things, because they do two different jobs.
It's really very simple.
This Fortran vs Python example is worth reading: https://cerfacs.fr/coop/fortran-vs-python
if you could import numpy==3.2 it'd solve many problems.
A depends on B, C
B depends on D==1.24
C depends on D==2.02
There should in an super ideal world be no problem with this. You would just have a “linker” that inserts itself in the module loader that presents different module objects when deps ask for “D” but it hasn’t happened yet.
- poetry
- "Just pin the dependencies and use Docker"
- pip freeze
- Vendoring in dependency code
- pipreqs
- virtualenv
This is simply a mess and it's handled much better in other languages. I manage a small agency team and there are some weeks where I feel like we need a full-time devops person to just help resolve environment issues with Python projects around the team.