News for hackers. Stuff that matters.
Re:Ask HN: Machine Learning Engineers, what do you do at work? by next_xibalba on Friday June 07, 2024 @02:52PM (#40611713)
Do people doing ML/DS not use conda anymore?
Re:Ask HN: Machine Learning Engineers, what do you do at work? by buildbot on Friday June 07, 2024 @02:59PM (#40611767)
A lot do, personally, every single time I try to go back to conda/mamba whatever, I get some extremely weird C/C++ related linking bug - just recently, I ran into an issue where the environment was _almost_ completely isolated from the OS distro's C/C++ build infra, except for LD, which was apparently so old it was missing the vpdpbusd instruction (https://github.com/google/XNNPACK/issues/6389). Except the thing was, that wouldn't happen when building outside of of the Conda environment. Very confusing. Standard virtualenvs are boring but nearly always work as expected in comparison.

I'm an Applied Scientist vs. ML Engineer, if that matters.

Re:Ask HN: Machine Learning Engineers, what do you do at work? by astromaniak on Friday June 07, 2024 @10:34PM (#40614847)
It's probably easier to reinstall everything anew from time to time. Instead of fixing broken 18.04 just move to 22.04. Most tools should work, if you don't have huge codebase which requires old compiler...

Conda.. it interfere with OS setup and has not always the best utils. Like ffmpeg is compiled with limited options, probably due to licensing.

Re:Ask HN: Machine Learning Engineers, what do you do at work? by buildbot on Saturday June 08, 2024 @02:10AM (#40615638)
I do all the time, and always have (in fact my first job was bare metal OS install automation), this was Rocky 9.4. New codebase, new compiler weird errors. I did actually reinstall and switch over to Ubuntu 24.04 after that issue lol.
Re:Ask HN: Machine Learning Engineers, what do you do at work? by copperroof on Friday June 07, 2024 @11:51PM (#40615134)
If they are they should stop.

It causes so many entirely unnecessary issues. The conda developers are directly responsible for maybe a month of my wasted debugging time. At my last job one of our questions for helping debug client library issues was “are you using conda”. And if so we just would say we can’t help you. Luckily it was rare, but if conda was involved it was 100% conda fault somehow, and it was always a stupid decision they made that flew in the face of the rest of the python packaging community.

Data scientist python issues are often caused by them not taking the 1-3 days it takes to fully understand their tool chain. It’s genuinely quite difficult to fuck up if you take the time once to learn how it all works, where your putbon binaries are on your system etc. Maybe not the case 5 years ago. But today it’s pretty simple.

Re:Ask HN: Machine Learning Engineers, what do you do at work? by buildbot on Saturday June 08, 2024 @02:12AM (#40615644)
Fully agree with this. Understand the basic tools that currently exist and you'll be fine. Conda constantly fucks things up in weird hard to debug ways...
Re:Ask HN: Machine Learning Engineers, what do you do at work? by blt on Saturday June 08, 2024 @02:39AM (#40615738)
I can't point at a single reason, but I got sick of it.

The interminable solves were awful. Mamba made it better, but can still be slow.

Plenty of more esoteric packages are on PyPI but not Conda. (Yes, you can install pip packages in a conda env file.)

Many packages have a default version and a conda-forge version; it's not always clear which you should use.

In Github CI, it takes extra time to install.

Upon installation it (by default) wants to mess around with your .bashrc and start every shell in a "base" environment.

It's operated by another company instead of the Python Software Foundation.

idk, none of these are deal-breakers, but I switched to venv and have not considered going back.