Oh my! This hits home. We have some test scripts written in python. Every time I try to run them after a few months I spend a day fixing the environment, package dependencies and other random stuff. Python is pretty nice once it works, but managing the environment can be a pain.

Can recommend using conda, more specifically mambaforge/micromamba (no licensing issues when used at work).

This works way better than pip, as it does more checks/dependency checking, so it does not break as easily as pip, though this makes it definitely way slower when installing something. It also supports updating your environment to the newest versions of all packages.

It's no silver bullet and mixing it with pip leads to even more breakages, but there is pixi [0] which aims to support interop between pypi and conda packages

[0] https://prefix.dev/

I had a bad experience with Conda:

- If they're so good at dependency management, why is Conda installed through a magical shell script?

- It's slow as molasses.

- Choosing between Anaconda/Miniconda...

When forced to use Python, I prefer Poetry, or just pip with freezing the dependencies.

The Python people probably can't even imagine how great dependency management is in all the other languages...

I absolutely hate conda. I had to support a bunch of researchers who all used it and it was a nightmare.

Mamba/micromamba solves the slowness problem of conda

To add. Conda has parallelized downloads and is faster. Not as fast as mamba, but faster than previously. pr merged sep 2022 -> https://github.com/conda/conda/pull/11841

Yeah, I agree, maybe I should have also mentioned the bad things about it, but after trying many different tools that's the one that I stuck with, as creating/destroying environments is a breeze once you got it working and the only time my environment broke was when I used pip in that environment.

> The Python people probably can't even imagine how great dependency management is in all the other languages...

Yep, I wish I could use another language at work.

> Choosing between Anaconda/Miniconda...

I went straight with mamba/micromamba as anaconda isn't open source.

rye and uv, while "experimental", are orders of magnitude better than poetry and pip IMHO.

Yes I started with conda I think and ended up switching to venv and I can’t even remember why, it’s a painful blur now. It was almost certainly user error too somewhere along the way (probably one of the earlier steps), but recovering from it had me seriously considering buying a Linux laptop.

This happened about a week ago

Whenever anyone recommends conda I automatically assume they don't know much about python.

If you're still doing ML locally in 2024 and also use an ARM macbook, you're asking for trouble.

> ARM macbook

Funnily, the only real competitor for Nvidias' GPUs are Macbooks with 128GB of RAM.

And they don't compete in performance.

I see your contemporary hardware choices and raise you my P900 ThinkStation with 256GB of RAM and 48 Xeon cores. Eventually it might even acquire modern graphics hardware.

Can you expand on this a bit? My recent experiences with MLX have been really positive, so I'm curious what footguns you're alluding to here.

(I don't do most of my work locally, but for smaller models its pretty convenient to work on my mbp).

MPS implementations generally lag behind CUDA kernels, especially for new and cutting edge stuff. Sure, if you're only running CPU inference or only want to use the GPU for simple or well established models, then things have gotten to the point where you can almost get the plug and play experience on Apple silicon. But if you're doing research level stuff and training your own models, the hassle is just not worth it once you see how convenient ML has become in the cloud. Especially since you don't really want to store large training datasets locally anyways.

For real

I wish my company would understand this and let us use something else. Luckily, they don't really seem to care that I use my Linux based gaming machine most of the time

What can I say, I enjoy pain!?

Nahh im l33t - intel macbook and no troubles.

can't read? parent clearly says "ARM macbook".

I thought most ML engineers use their laptops as dumb terminals and just remote into a Linux GPU server.

Yeah, the workday there looks pretty similar though, except that installing pytorch and pillow is usually no problem. Today it was flash-attn I spent the afternoon on.

Isn't this what containers are for. Someone somewhere gets it configured right and then you download and run pre setup container and add your job data ? Or am I looking at the problem wrong?

But then how do you test out the latest model that came out from who knows where and has the weirdest dependencies and a super obscure command to install?

Just email all your data to the author and ask them to run it for you.

Spoiler: my main role isn’t ML engineer :) and that doesn’t sound like a bad idea at all

I do this... but air-gapped :(

Oof. At our company only CI/CD agents (and laptops) are allowed to access the internet, and that's bad enough.

that sounds very painful.

Do people doing ML/DS not use conda anymore?

A lot do, personally, every single time I try to go back to conda/mamba whatever, I get some extremely weird C/C++ related linking bug - just recently, I ran into an issue where the environment was _almost_ completely isolated from the OS distro's C/C++ build infra, except for LD, which was apparently so old it was missing the vpdpbusd instruction (https://github.com/google/XNNPACK/issues/6389). Except the thing was, that wouldn't happen when building outside of of the Conda environment. Very confusing. Standard virtualenvs are boring but nearly always work as expected in comparison.

I'm an Applied Scientist vs. ML Engineer, if that matters.

It's probably easier to reinstall everything anew from time to time. Instead of fixing broken 18.04 just move to 22.04. Most tools should work, if you don't have huge codebase which requires old compiler...

Conda.. it interfere with OS setup and has not always the best utils. Like ffmpeg is compiled with limited options, probably due to licensing.

I do all the time, and always have (in fact my first job was bare metal OS install automation), this was Rocky 9.4. New codebase, new compiler weird errors. I did actually reinstall and switch over to Ubuntu 24.04 after that issue lol.

If they are they should stop.

It causes so many entirely unnecessary issues. The conda developers are directly responsible for maybe a month of my wasted debugging time. At my last job one of our questions for helping debug client library issues was “are you using conda”. And if so we just would say we can’t help you. Luckily it was rare, but if conda was involved it was 100% conda fault somehow, and it was always a stupid decision they made that flew in the face of the rest of the python packaging community.

Data scientist python issues are often caused by them not taking the 1-3 days it takes to fully understand their tool chain. It’s genuinely quite difficult to fuck up if you take the time once to learn how it all works, where your putbon binaries are on your system etc. Maybe not the case 5 years ago. But today it’s pretty simple.

Fully agree with this. Understand the basic tools that currently exist and you'll be fine. Conda constantly fucks things up in weird hard to debug ways...

I can't point at a single reason, but I got sick of it.

The interminable solves were awful. Mamba made it better, but can still be slow.

Plenty of more esoteric packages are on PyPI but not Conda. (Yes, you can install pip packages in a conda env file.)

Many packages have a default version and a conda-forge version; it's not always clear which you should use.

In Github CI, it takes extra time to install.

Upon installation it (by default) wants to mess around with your .bashrc and start every shell in a "base" environment.

It's operated by another company instead of the Python Software Foundation.

idk, none of these are deal-breakers, but I switched to venv and have not considered going back.

Maybe pip should not work by default (but python -m venv then pip install should)

Legends say there were times when you'd have a program.c file and just run cc program.c, and then could just execute the compiled result. Funny that programmer's job is highly automatable, yet we invent ourselves tons of intermediate layers which we absolutely have to deal with manually.

And then you'd have to deal with wrong glibc versions or mysterious segfaults or undefined behavior or the the code assuming the wrong arch or ...

python solves none of those issues. It just adds a myriad of ways those problems can get to you.

All of a sudden you have people with C problems, who have no idea they're even using compiled dependencies.

In theory you're right, CPython is written in C and it could segfault or display undefined behavior. In practice, you're quite wrong.

It's not really much of a counterargument to say that Python is good enough that you don't have to care what's under the hood, except when it breaks because C suck so badly.

I was specifically talking about python packages using C. You type "pip install" and god knows what's going to happen. It might pull a precompiled wheel, it might just compile and link some C or Fortran code, it might need external dependecies. It might install flawlessly and crash as soon as you try to run it. All bets are off.

I never experienced CPython itself segfault, it's always due to some package.

I actually did a small C project a couple of years ago, the spartan simplicity there can have its own pain too, like having to maintain a Makefile. LOL. It’s swings and roundabouts!

I agree simplicity is king. But you're comparing making a script using dependencies and tooling for those dependencies and a C program with no dependencies. You can download a simple python script and run it directly if it has no dependencies besides stdlib (which is way larger in python). That's why I love using bottle.py by example.

Agree. But even with dependencies running "make" seems to be way simpler than having to install particular version of tools for a project, making venv and then picking versions of dependencies.

The point is the same - we had it simpler and now, with all capabilities for automation, we have it more complex.

Frankly, I suspect most of the efforts now are spent fighting non-essential complexities, like compatibilities, instead of solving the problem at hand. That means we create problems for ourselves faster than removing them.

You can do the same in python if you're not new to programming.

Some Linux distros are moving that way, particularly for the included Python/pip version. My Arch Linux already does so some years, and I do not set it up myself - so I think it is default.

As an amateur game engine developer, I morosely reflect my hobby seems to actually consist of endlessly chasing things that were broken by environment updates (OS, libraries, compiler, etc.) That is, most of the time I sit down to code I actually spend nuking and reinstalling things that (I thought) were previously working.

Your comment makes me feel a little better that this is not merely some personal failing of focus, but happens in a professional setting too.

Happens in AAA too but we tend to have teams that shield everyone from that before they get to work. I ran a team like that for a couple years.

For hobby stuff at home though I don't tend to hit those types of issues because my projects are pretty frozen dependency-wise. Do you really have OS updates break stuff for you often? I'm not sure I recall that happening on a home project in quite a while.

Oh god yes I remember trying to support old Android games several OS releases later… Impossible, I gave up!

It’s why I still use react, their backcompat is amazing

Docker is your friend

or pyenv at least

Yes, I’ve switched from conda to a combination of dev containers and pyenv/pyenv-virtualenv on both my Linux and MacBook machines and couldn’t be happier

Just use conda.

Then you get one of my favorites: NVIDIA-<something> has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Iirc I originally used conda because I couldn’t get faiss to work in venv, lol. That was a while ago though

Docker fixes this!

> Environment broken

>Something something incorrect cpu architecture for your Macbook

I’m glad I have something in common with the smart people around here.

Yeah getting a good python environment setup is a very humbling experience

uv is a great drop-in replacement for pip: https://astral.sh/blog/uv

Why do ML, especially nowadays on a Mac, when you can do it on an Ubuntu based machine.

Surely work can provide that?

Why Ubuntu specifically? Not even being snarky. Calling out a specific distro, vs. the operating system itself. I've had more pain setting up ML environments with Ubuntu than a MacBook, personally - though pure Debian has been the easiest to get stable from scratch. Ubuntu usually screws me over one way or another after a month or so. I think I've spend a cumulative month of my life tracking down things related to changes inn netplan, cloud-init, etc. Not to mention Ubuntu Pro spam being incessant, as official policy of Canonical [0]. I first used the distro all the way back at Warty Warthog, and it was my daily driver from Feisty until ~Xenial. I think it was the Silicon Valley ad in the MotD that was the last straw for me.

[0] https://bugs.launchpad.net/ubuntu/+source/ubuntu-meta/+bug/1...

I can recommend to try poetry. It is a lot more succesful in resolving dependencies than pip.

Although I think the UX of poetry is stupid and I do not agree with some design decisions, I have not had any dependency conflicts since I used it.

You could learn how to use Python. Just spend one of those 4 hours actually learning. Imagine just getting into a car and pressing controls until something happened. This wouldn't be allowed to happen in any other industry.

Could you be a bit more specific about what you mean by "You could learn how to use python"? What resources would you recommend to learn how to work around problems the OP has? What basic procedures/resources can you recommend to "learn python"? I work as a software developer alongside my studies and often face the same problems as OP that I would like to avoid. Very grateful for any tips!

Basically just use virtual environments via the venv module. The only thing you really need to know is that Python doesn't support having multiple versions of a package installed in the same environment. That means you need to get very familiar with creating (and destroying) environments. You don't need to know any of this if you just use tools that happen to be written in Python. But if you plan to write Python code then you do. It should be in Python books really, but they tend to skip over the boring stuff.

How do you learn anything in the space of software engineering? In general, there are many different problems and even more solutions with different tradeoffs.

To avoid spending hours on fixing broken environments after a single "pip install", I would make it easy to rollback to a known state. For example, recreate virtualenv from a lock requirements file stored in git: `pipenv sync` (or corresponding command using your preferred tool).

Oh wow, I’ve been a Python engineer for over a decade and getting dependencies right for machine learning has very little to do with Python and everything to do with c++/cuda

I've done it. Isn't it just following instructions? What part of that means destroying every mention of Python on the system?

I've been programming with python for decades and the problem they are describing says more about the disastrous state of python's package management and the insane backwards compatibility stance python devs have.

Half of the problems I've helped some people solve stem from python devs insisting on shuffling std libraries around between minor versions.

Some libraries have a compatibility grid with different python minor versions, because how often they break things.

...and people criticise node's node_modules. At least you don't spend hours doing this

But you do because your local node_modules and upstream are out of sync and CI is broken. Happens at least once a month just before a release of course. I'd rather have my code failing locally than trying to debug what's out of sync on upstream.

Not nearly as hard of a problem. Python does work just fine when it’s pure Python. The trouble comes with all the C/Cuda dependencies in machine learning

Python's dominance is holding us back. We need a stack with a more principled approach to environments and native dependencies.

Here's what getting PyTorch built reproducibly looks like: https://hpc.guix.info/blog/2021/09/whats-in-a-package/

Since then the whole python ecosystem has gotten worse.

We are building towers on quicksand.

It's not about python, it's about people who don't care about dependencies.

Dependency management is just.. hard. It is one of the things where everything relies upon it but nobody thinks "hey, this is my responsibility to improve" so it is left to people who have the most motivation, academic posts, or grant funding. This is roughly the same problem that led to heartbleed for OpenSSL.

Do you know what other ecosystem comes closest to the existing in Python? I've heard good things about Julia.

13 years ago when I was trying to explore the field R seemed to be the most popular, but looks like not anymore. (I didn't get into the field, and do just a regular SWE, so I'm not aware of the trends).

There is also a lot of development in Elixir ecosystem around the subject [1].

[1](https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instruc...)

I don't think the ML community has an appetite for learning a different language.

There are Microsoft-backed F# bindings for Spark and Torch, but no one seems interested. And this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance.

The answer will probably be JavaScript.

Everyone already knows the language - all that's missing is operator overloading and a few key bindings.

> There are Microsoft-backed F# bindings for Spark and Torch > this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance.

For exactly the reasons your mentioned, i feel like F# would have been the perfect match for both MLE/ETL(spark pipeline) work and some of the deep learning/graph modelisation such as pytorch. Saddly, even from MSFT, the investment in F# as dried up

Not sure what you mean here - Microsoft are actively working on F# and the .NET Spark and Torch bindings.

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc.

Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist?

Are python engineers just bad, or?

(Background: I never used python except for one time when I took a coursera ML course and was immediately assaulted with conda/miniconda/venv/pip/etc etc and immediately came away with a terrible impression of the ecosystem.)

Two problems intersect:

- You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility

This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above.

There is no good solution to the above problem.

This problem is amplified when lots of the packages were written by academics 10 years ago and are no longer maintained.

The bad solutions are: 1) Have 2 venvs - not always possible and if you keep making venvs you’ll have loads of them. 2) Rewrite your code to only use one library 3) Update one of the libraries 4) Don’t care about the mismatch and cross your fingers that the old one will work with the newer library.

Most of the tooling follows approach 1 or 4

Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv.

I threw it online at https://github.com/fragmede/python-wool/

You’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions.

Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE

Yeah that sucks. something like:

    for venv in $(find ~/projects/ -type d -name 'venv'); do
      (
        source ${venv}/bin/activate
        pip install --upgrade pip
        pip install --upgrade package_with_cve
        deactivate
      ) &
    done

should do the trick. Sucks that we're in that world, but let's not work any harder than we have to.

I forgot about parallel, which would do better than spawning off all the pip install --upgrade at once.

After many years of Django development I've settled on what I find the simplest solution. It includes activation of virtual environments when I cd into a directory.

pyenv for installing and managing python versions.

direnv for managing environments and environment variables (a highly underrated package imo).

With those two installed I just include a .envrc file in every project. It looks like this:

layout python ~/.pyenv/versions/3.11.0/bin/python3

export VARIABLE1=variable1

export VARIABLE2=variable2

etc...

Oh nice. Stick

    PYTHONPATH="`pwd`/venv/lib/python-3.11/site-packages"

or whatever in .envrc and Bob's your uncle.

I think it is worth separating the Python ML ecosystem from the rest. While traditional Python environment management has many sore points, it is usually not terrible (though there are many gotchas still-to-this-day-problems that should have been corrected long ago).

The ML system is a whole another stack of problems. The elephant in the room is Nvidia who is not known for playing well with others. Aside from that, the state of the art in ML is churning rapidly as new improvements are identified.

It's not bad. It works really well. There's always room for improvement. That's technology for you. Python probably does attract more than it's fair share of bad engineers, though.

Unironically yes, it really is that bad. A moderately bad language that happened to have some popular data science and math libraries from the beginning.

I can only imagine it seemed like an oasis to R which is bottom tier.

So when you combine data scientists, academics, mathematicians, juniors, grifters selling courses…

things like bad package management, horrible developer experience, absolutely no drive in the ecosystem to improve anything beyond the “pinnacle” of wrapping C libraries are all both inevitable and symptoms of a poorly designed ecosystem.

Just replace python with mac or apple in your comment and I think you will understand.

Just use Linux. Then you only have to fight the nvidia drivers.

Use standard cloud images

> Something something incorrect cpu architecture for your Macbook

Really can't recommend Nix for Python stuff more, it's by far the best at managing dependencies that use FFI/native extensions. It can be a pain sometimes to port your existing Poetry/etc. project to work with nix2lang converters but really pays off.

So what I do in my free time, but getting paid for it?? Sign me up.