News for hackers. Stuff that matters.
Re:PyTorch 2.0 Release by mardifoufs on Wednesday March 15, 2023 @06:22PM (#35175538)
>Python 3.11 support on Anaconda Platform

>Due to lack of Python 3.11 support for packages that PyTorch depends on, including NumPy, SciPy, SymPy, Pillow and others on the Anaconda platform. We will not be releasing Conda binaries compiled with Python 3.11 for PyTorch Release 2.0. The Pip packages with Python 3.11 support will be released, hence if you intend to use PyTorch 2.0 with Python 3.11 please use our Pip packages.

It really sucks that anaconda always lags behind. I know the reasoning*, and I know it makes sense for what a lot of teams use it for... but on our side we are now looking more and more into dropping it since we are more of an R&D team. We already use containers for most of our pipelines, so just using pip might be viable.

*Though I guess Anaconda chewed more than it can handle w.r.t managing an entire Python universe, and keeping up to date. Conda-forge is already almost a requirement but using the official package (with pip, in this case) has its own benefits for very complex packages like pytorch.

Re:PyTorch 2.0 Release by brucethemoose2 on Wednesday March 15, 2023 @06:37PM (#35175677)
The Arch Linux PyTorch 2.0 packages are great if you are looking for "cutting edge," as they are compiled against CUDA 12.1 now, instead of 11.8 like the official nightly releases. You can also get AVX2 patched Python and optimized C Python packages through CachyOS or ALHP.

But even Arch is still stuck on Python 3.10

Re:PyTorch 2.0 Release by DreamFlasher on Wednesday March 15, 2023 @06:45PM (#35175747)
Afaik NumPy, SciPy, SymPy and Pillow are not managed/owned by Anaconda? At least here: https://numpy.org/about/ Anaconda isn't mentioned.
Re:PyTorch 2.0 Release by DreamFlasher on Wednesday March 15, 2023 @06:46PM (#35175764)
Ah, yeah they do have a Python 3.11 release, just not on anaconda. Okay, yeah, for a couple of years now there isn't a good reason anymore to use anaconda anyways.
Re:PyTorch 2.0 Release by mardifoufs on Wednesday March 15, 2023 @06:50PM (#35175795)
Yes that's the issue! Most of the software is already ready, usable and just works... unless you use anaconda. Now that I think about it, is there some technical reason for that? I always thought it was mostly about stability, but I can't imagine python 3.11 being so unstable as to warrant waiting a whole year before even porting.
Re:PyTorch 2.0 Release by PaulHoule on Wednesday March 15, 2023 @08:28PM (#35176593)
For one of my projects, conda ‘just works’ to get working with the GPU but following he instructions for pip doesn’t work. On the other hand there’s another package I am interested in using where I need to build out of GitHub and it’s a very different story.

I see myself as interested in commercial exploitation of transformers right now and I am delighted with the results. The first time I tried clustering all the Ukraine articles lumped together, all the sports were lumped, it runs 5x faster than my LDA-based clustering system and I think does a better job. With results like this I am happy to trade ‘cutting edge’ for convenience.

I have thought about a ‘path less followed’ in Python which is a truly sound package manager like maven for Python (as opposed to Poetry which I’m not sure is sound but it sure is slow) and I can say I like the way conda works I just would rather do it with wheels. One beef I have with conda is that the bzip2 files are slow to decompress and even over a DSL line I would trade a little more downloading for faster installs.

Re:PyTorch 2.0 Release by Vox_Leone on Wednesday March 15, 2023 @11:07PM (#35177673)
>>for a couple of years now there isn't a good reason anymore to use anaconda anyways.

I think of Anaconda envs as great back ends for Jupyter notebooks. There is still a place for it.

Re:PyTorch 2.0 Release by hmaarrfk on Wednesday March 15, 2023 @08:19PM (#35176520)
All the projects you listed are community projects that have grown over the years to become the backbone of scientific computing.

conda-forge, has evolved as one of the major conda community projects that helps release the latest releases from the projects you listed.

You can help improve the state of pytorch packaging on conda-forge too!

We've even released Pytorch 1.13 + Python 3.11 on linux and OSX! Give it a shot and let us know what you think!

edit: Link to the conda-forge pytorch development repository https://github.com/conda-forge/pytorch-cpu-feedstock

Re:PyTorch 2.0 Release by varispeed on Wednesday March 15, 2023 @07:04PM (#35175899)
Why people use conda? I couldn't figure out what it brings to the table.

I use pyenv on Linux and Mac and it basically works and with virtualenv plugin I can quickly switch environments to different Python versions.

Re:PyTorch 2.0 Release by aldanor on Wednesday March 15, 2023 @07:36PM (#35176191)
- Managing non-python / binary dependencies alongside python ones

- Managing a mixture of python / binary dependencies in local registries

- conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally

Re:PyTorch 2.0 Release by nl on Wednesday March 15, 2023 @08:05PM (#35176406)
These are all legitimate reasons, but my personal experience (and perhaps preference?) is to use Docker for anything that is more complex than pip can handle.

> conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally

Yeah this is fair. Fortunately it's becoming rarer.

Re:PyTorch 2.0 Release by PaulHoule on Wednesday March 15, 2023 @08:32PM (#35176641)
Seems docker is going from the frying pan to the fire. Have they added ‘resume download’ yet to docker? Over my slow DSL I can’t stand how docker makes me download 7G of images when I want to install something very simple, frequently it fails to do the download and I have to do it several times so it adds up to more like 28G of downloading and all that waiting.

I worked at one place where management was shocked when I told them the image build process would take 20 minutes on gigabit fiber up in Canada and we agreed to time it and I measured 18 minutes. Docker slows down “dev” to the speed of “ops.”

I don’t know how they did it but the data scientists could always find f-ed up Python images, you never got the same default character encoding twice, one time the default character was for Hungarian and I wonder how that happens…

Re:PyTorch 2.0 Release by aldanor on Wednesday March 15, 2023 @08:11PM (#35176463)
Well. To each their own, use cases differ so wildly it's hard to compare them.

The key audience for conda is ML/DS space, where most if not all packages come from either C/C++/Rust/Fortran and have to be compiled, while also requiring a consistent set of external C libraries like libblas, etc. As I said, some of those packages are a completely nightmare to build locally. Conda simplifies this by a lot in that you can just 'conda create -n myenv some=1.0 crazy=2.0 deps=2.0' and in a few seconds (if you use mamba and not conda) you have a working Python environment so off you go; no dockers, no local builds etc.

Re:PyTorch 2.0 Release by techwizrd on Wednesday March 15, 2023 @08:56PM (#35176813)
Honestly, I've found that conda has made operationalizing code very difficult. We've found it much easier to simply switch back to using pip, poetry, docker, and the standard OS package management tools rather than conda. Conda's dependency resolution is also quite slow and causes our builds & CI to timeout unless we drop in mamba.
Re:PyTorch 2.0 Release by nl on Thursday March 16, 2023 @01:41AM (#35178541)
Yes, precisely this. (I only do ML/DS work)
Re:PyTorch 2.0 Release by thrdbndndn on Wednesday March 15, 2023 @10:55PM (#35177597)
Can these hard-to-build-locally lib just upload a wheel to PYPI?
Re:PyTorch 2.0 Release by hobs on Wednesday March 15, 2023 @11:11PM (#35177703)
lol, the dependency hell of even using mamba can be large on old things, I have waited more than an hour for dependency resolution using both.
Re:PyTorch 2.0 Release by drdaeman on Thursday March 16, 2023 @01:46AM (#35178573)
Doesn't pip (with wheels) solve this all just fine? Or I'm missing something?
Re:PyTorch 2.0 Release by silveraxe93 on Thursday March 16, 2023 @05:51AM (#35179972)
You're missing something. - Managing non-python / binary dependencies alongside python ones

pip with wheels doesn't deal with non-python packages. I used to be in a horrible locked down corpo laptop. Conda was invaluable in getting stuff to run, like chromedriver, etc.

Re:PyTorch 2.0 Release by drdaeman on Thursday March 16, 2023 @02:23PM (#35186091)
Uh, I guess I do? I mean, there is https://pypi.org/project/chromedriver-binary/ (and a lot of similar packages)...

I honestly don't remember which one I've used for chromedriver when I needed it for my project, but I've surely installed all the stuff with "just" pip/poetry. Larger projects are typically packaged like this, with setup.py performing the downloads, while wheels solve the problem with Python libraries with native dependencies (e.g. how psycopg-binary works).

Maybe Conda makes it slightly more convenient, but I've always treated pip as the standard Python package management tool (it's a part of the standard library now, after all) and Conda was always "that weird non-standard thing some folks use for some odd reason" for me.

Re:PyTorch 2.0 Release by DethNinja on Wednesday March 15, 2023 @07:30PM (#35176126)
I find it far better for DevOps. It is really easy to get the team up and running via conda environment files.
Re:PyTorch 2.0 Release by est on Thursday March 16, 2023 @02:51AM (#35178956)
> Why people use conda? I couldn't figure out what it brings to the table.

Have you tried compile mkl, numpy from scratch?

Re:PyTorch 2.0 Release by kzrdude on Thursday March 16, 2023 @06:26AM (#35180151)
Pip is a good alternative for numpy, it doesn't need conda. I think mkl is the same.
Re:PyTorch 2.0 Release by raverbashing on Thursday March 16, 2023 @09:08AM (#35181414)
pip install numpy works

(though using poetry makes things less hectic in case of upgrades)

Re:PyTorch 2.0 Release by vimy on Wednesday March 15, 2023 @09:04PM (#35176869)
I use anaconda environments and install everything with pip in them. Am I doing it wrong?
Re:PyTorch 2.0 Release by blandcoffee on Wednesday March 15, 2023 @09:36PM (#35177084)
What's the point of this vs. creating your own venv (python -m venv .venv) and having a much smaller library of intentional tools?
Re:PyTorch 2.0 Release by tagh on Wednesday March 15, 2023 @10:14PM (#35177346)
Each environment can have its own version of python which downloads with everything else (venv would require separate system installs for this).
Re:PyTorch 2.0 Release by blandcoffee on Wednesday March 15, 2023 @11:06PM (#35177671)
Can you clarify what you mean by separate system installs?

Today, I manage python via pyenv (local and global state) that let's me create infinite .venvs with different python versions.

Re:PyTorch 2.0 Release by nerdponx on Thursday March 16, 2023 @10:57AM (#35182796)
Conda does the same job as Pyenv here, with the additional ability to install system-level libraries, compiler toolchains, etc.
Re:PyTorch 2.0 Release by inconceivable on Wednesday March 15, 2023 @11:56PM (#35177973)
i use venv this way. i download and compile specific python versions and install them in a non-system dir with all the other versions. then just run the specific binary to create a venv and it seems to work as expected.
Re:PyTorch 2.0 Release by alexchantavy on Thursday March 16, 2023 @12:02AM (#35178008)
Anaconda automatically handles things like making sure the correct version of cuDNN for your graphics card is installed. When I tried doing this myself with venv it was really painful.
Re:PyTorch 2.0 Release by mardifoufs on Wednesday March 15, 2023 @09:28PM (#35177028)
That's basically where we are at for tons of our pipelines, but it kind of defeats the purpose since a dockerfile with a proper base image is basically equivalent at that point.
Re:PyTorch 2.0 Release by disgruntledphd2 on Thursday March 16, 2023 @05:53AM (#35179985)
Conda is also really annoying to get working in a Dockerfile, because (if I remember correctly) of differences between login and non-login shells.
Re:PyTorch 2.0 Release by nerdponx on Thursday March 16, 2023 @10:59AM (#35182809)
In general you shouldn't need to "activate" a Conda environment in the shell. Things generally "just work" if you use absolute paths. Something like this:

  COPY conda.yaml /env/conda.yaml
  RUN conda env create -f /env/conda.yaml -p /env/conda

  COPY foo.py /env/foo.py
  CMD ["/env/conda/bin/python", "/env/foo.py"]
What is a little funny is installing a consistent version of Conda inside a container, because the official Miniconda installers are rolling-release only. However you might be able to downgrade to your desired version of Conda after installation.
Re:PyTorch 2.0 Release by nerdponx on Thursday March 16, 2023 @10:56AM (#35182790)
There's nothing wrong with this. IMO, Conda is a general-purpose "system environment" and package manager that happens to be written in Python. The fact that its package ecosystem is oriented towards machine learning with Python is almost an historical coincidence.
Re:PyTorch 2.0 Release by vishal0123 on Thursday March 16, 2023 @07:56PM (#35190689)
Exactly my setup. I tried to use `conda install` few times, but every time after just few globally installed packages, conda SAT solver always struggles, and I now live with assumption that if incompatible package combination does not throw any error in dev environment, it is likely fine.
Re:PyTorch 2.0 Release by alephxyz on Friday March 17, 2023 @10:44AM (#35197767)
FYI the libmamba solver released last year is way faster than the classic one at modifying environments.
Re:PyTorch 2.0 Release by dheera on Wednesday March 15, 2023 @10:01PM (#35177250)
> It really sucks that anaconda always lags behind.

I usually just go for virtualenv (if python library versions are the only issue) or go for docker (if it's more than that). Both let you just use the latest and greatest without any friction. conda sits in a weird middle ground that I hate.

Re:PyTorch 2.0 Release by disgruntledphd2 on Thursday March 16, 2023 @05:52AM (#35179975)
> *Though I guess Anaconda chewed more than it can handle w.r.t managing an entire Python universe, and keeping up to date. Conda-forge is already almost a requirement but using the official package (with pip, in this case) has its own benefits for very complex packages like pytorch.

Yeah, I absolutely adore conda, but they really need support.

Re:PyTorch 2.0 Release by urthor on Thursday March 16, 2023 @07:11AM (#35180473)
Anaconda is really a solution for a personal dev box yes.

For containers a version controlled requirements.txt or Dockerfile is the way.

Realistically, Anaconda is a venerable solution from the truly god awful old days. The ecosystem is vastly improved.