News for hackers. Stuff that matters.
Re:PyTorch 2.0 Release by vimy on Wednesday March 15, 2023 @09:04PM (#35176869)
I use anaconda environments and install everything with pip in them. Am I doing it wrong?
Re:PyTorch 2.0 Release by blandcoffee on Wednesday March 15, 2023 @09:36PM (#35177084)
What's the point of this vs. creating your own venv (python -m venv .venv) and having a much smaller library of intentional tools?
Re:PyTorch 2.0 Release by tagh on Wednesday March 15, 2023 @10:14PM (#35177346)
Each environment can have its own version of python which downloads with everything else (venv would require separate system installs for this).
Re:PyTorch 2.0 Release by blandcoffee on Wednesday March 15, 2023 @11:06PM (#35177671)
Can you clarify what you mean by separate system installs?

Today, I manage python via pyenv (local and global state) that let's me create infinite .venvs with different python versions.

Re:PyTorch 2.0 Release by nerdponx on Thursday March 16, 2023 @10:57AM (#35182796)
Conda does the same job as Pyenv here, with the additional ability to install system-level libraries, compiler toolchains, etc.
Re:PyTorch 2.0 Release by inconceivable on Wednesday March 15, 2023 @11:56PM (#35177973)
i use venv this way. i download and compile specific python versions and install them in a non-system dir with all the other versions. then just run the specific binary to create a venv and it seems to work as expected.
Re:PyTorch 2.0 Release by alexchantavy on Thursday March 16, 2023 @12:02AM (#35178008)
Anaconda automatically handles things like making sure the correct version of cuDNN for your graphics card is installed. When I tried doing this myself with venv it was really painful.
Re:PyTorch 2.0 Release by mardifoufs on Wednesday March 15, 2023 @09:28PM (#35177028)
That's basically where we are at for tons of our pipelines, but it kind of defeats the purpose since a dockerfile with a proper base image is basically equivalent at that point.
Re:PyTorch 2.0 Release by disgruntledphd2 on Thursday March 16, 2023 @05:53AM (#35179985)
Conda is also really annoying to get working in a Dockerfile, because (if I remember correctly) of differences between login and non-login shells.
Re:PyTorch 2.0 Release by nerdponx on Thursday March 16, 2023 @10:59AM (#35182809)
In general you shouldn't need to "activate" a Conda environment in the shell. Things generally "just work" if you use absolute paths. Something like this:

  COPY conda.yaml /env/conda.yaml
  RUN conda env create -f /env/conda.yaml -p /env/conda

  COPY foo.py /env/foo.py
  CMD ["/env/conda/bin/python", "/env/foo.py"]
What is a little funny is installing a consistent version of Conda inside a container, because the official Miniconda installers are rolling-release only. However you might be able to downgrade to your desired version of Conda after installation.
Re:PyTorch 2.0 Release by nerdponx on Thursday March 16, 2023 @10:56AM (#35182790)
There's nothing wrong with this. IMO, Conda is a general-purpose "system environment" and package manager that happens to be written in Python. The fact that its package ecosystem is oriented towards machine learning with Python is almost an historical coincidence.
Re:PyTorch 2.0 Release by vishal0123 on Thursday March 16, 2023 @07:56PM (#35190689)
Exactly my setup. I tried to use `conda install` few times, but every time after just few globally installed packages, conda SAT solver always struggles, and I now live with assumption that if incompatible package combination does not throw any error in dev environment, it is likely fine.
Re:PyTorch 2.0 Release by alephxyz on Friday March 17, 2023 @10:44AM (#35197767)
FYI the libmamba solver released last year is way faster than the classic one at modifying environments.