Re:PyTorch 2.0 Release
by vimy on Wednesday March 15, 2023 @09:04PM
(#35176869)
I use anaconda environments and install everything with pip in them. Am I doing it wrong?
[ Parent
| on HN ]
Re:PyTorch 2.0 Release
by blandcoffee on Wednesday March 15, 2023 @09:36PM
(#35177084)
What's the point of this vs. creating your own venv (python -m venv .venv) and having a much smaller library of intentional tools?
Each environment can have its own version of python which downloads with everything else (venv would require separate system installs for this).
Re:PyTorch 2.0 Release
by blandcoffee on Wednesday March 15, 2023 @11:06PM
(#35177671)
Can you clarify what you mean by separate system installs?
Today, I manage python via pyenv (local and global state) that let's me create infinite .venvs with different python versions.
Conda does the same job as Pyenv here, with the additional ability to install system-level libraries, compiler toolchains, etc.
Re:PyTorch 2.0 Release
by inconceivable on Wednesday March 15, 2023 @11:56PM
(#35177973)
i use venv this way. i download and compile specific python versions and install them in a non-system dir with all the other versions. then just run the specific binary to create a venv and it seems to work as expected.
Re:PyTorch 2.0 Release
by alexchantavy on Thursday March 16, 2023 @12:02AM
(#35178008)
Anaconda automatically handles things like making sure the correct version of cuDNN for your graphics card is installed. When I tried doing this myself with venv it was really painful.
Re:PyTorch 2.0 Release
by mardifoufs on Wednesday March 15, 2023 @09:28PM
(#35177028)
That's basically where we are at for tons of our pipelines, but it kind of defeats the purpose since a dockerfile with a proper base image is basically equivalent at that point.
Re:PyTorch 2.0 Release
by disgruntledphd2 on Thursday March 16, 2023 @05:53AM
(#35179985)
Conda is also really annoying to get working in a Dockerfile, because (if I remember correctly) of differences between login and non-login shells.
In general you shouldn't need to "activate" a Conda environment in the shell. Things generally "just work" if you use absolute paths. Something like this:
COPY conda.yaml /env/conda.yaml
RUN conda env create -f /env/conda.yaml -p /env/conda
COPY foo.py /env/foo.py
CMD ["/env/conda/bin/python", "/env/foo.py"]
What is a little funny is installing a consistent version of Conda inside a container, because the official Miniconda installers are rolling-release only. However you might be able to downgrade to your desired version of Conda after installation.There's nothing wrong with this. IMO, Conda is a general-purpose "system environment" and package manager that happens to be written in Python. The fact that its package ecosystem is oriented towards machine learning with Python is almost an historical coincidence.
Re:PyTorch 2.0 Release
by vishal0123 on Thursday March 16, 2023 @07:56PM
(#35190689)
Exactly my setup. I tried to use `conda install` few times, but every time after just few globally installed packages, conda SAT solver always struggles, and I now live with assumption that if incompatible package combination does not throw any error in dev environment, it is likely fine.