But even Arch is still stuck on Python 3.10
I see myself as interested in commercial exploitation of transformers right now and I am delighted with the results. The first time I tried clustering all the Ukraine articles lumped together, all the sports were lumped, it runs 5x faster than my LDA-based clustering system and I think does a better job. With results like this I am happy to trade ‘cutting edge’ for convenience.
I have thought about a ‘path less followed’ in Python which is a truly sound package manager like maven for Python (as opposed to Poetry which I’m not sure is sound but it sure is slow) and I can say I like the way conda works I just would rather do it with wheels. One beef I have with conda is that the bzip2 files are slow to decompress and even over a DSL line I would trade a little more downloading for faster installs.
I think of Anaconda envs as great back ends for Jupyter notebooks. There is still a place for it.
conda-forge, has evolved as one of the major conda community projects that helps release the latest releases from the projects you listed.
You can help improve the state of pytorch packaging on conda-forge too!
We've even released Pytorch 1.13 + Python 3.11 on linux and OSX! Give it a shot and let us know what you think!
edit: Link to the conda-forge pytorch development repository https://github.com/conda-forge/pytorch-cpu-feedstock
I use pyenv on Linux and Mac and it basically works and with virtualenv plugin I can quickly switch environments to different Python versions.
- Managing a mixture of python / binary dependencies in local registries
- conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally
> conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally
Yeah this is fair. Fortunately it's becoming rarer.
I worked at one place where management was shocked when I told them the image build process would take 20 minutes on gigabit fiber up in Canada and we agreed to time it and I measured 18 minutes. Docker slows down “dev” to the speed of “ops.”
I don’t know how they did it but the data scientists could always find f-ed up Python images, you never got the same default character encoding twice, one time the default character was for Hungarian and I wonder how that happens…
The key audience for conda is ML/DS space, where most if not all packages come from either C/C++/Rust/Fortran and have to be compiled, while also requiring a consistent set of external C libraries like libblas, etc. As I said, some of those packages are a completely nightmare to build locally. Conda simplifies this by a lot in that you can just 'conda create -n myenv some=1.0 crazy=2.0 deps=2.0' and in a few seconds (if you use mamba and not conda) you have a working Python environment so off you go; no dockers, no local builds etc.
pip with wheels doesn't deal with non-python packages. I used to be in a horrible locked down corpo laptop. Conda was invaluable in getting stuff to run, like chromedriver, etc.
I honestly don't remember which one I've used for chromedriver when I needed it for my project, but I've surely installed all the stuff with "just" pip/poetry. Larger projects are typically packaged like this, with setup.py performing the downloads, while wheels solve the problem with Python libraries with native dependencies (e.g. how psycopg-binary works).
Maybe Conda makes it slightly more convenient, but I've always treated pip as the standard Python package management tool (it's a part of the standard library now, after all) and Conda was always "that weird non-standard thing some folks use for some odd reason" for me.
Have you tried compile mkl, numpy from scratch?
(though using poetry makes things less hectic in case of upgrades)
Today, I manage python via pyenv (local and global state) that let's me create infinite .venvs with different python versions.
COPY conda.yaml /env/conda.yaml
RUN conda env create -f /env/conda.yaml -p /env/conda
COPY foo.py /env/foo.py
CMD ["/env/conda/bin/python", "/env/foo.py"]
What is a little funny is installing a consistent version of Conda inside a container, because the official Miniconda installers are rolling-release only. However you might be able to downgrade to your desired version of Conda after installation.I usually just go for virtualenv (if python library versions are the only issue) or go for docker (if it's more than that). Both let you just use the latest and greatest without any friction. conda sits in a weird middle ground that I hate.
Yeah, I absolutely adore conda, but they really need support.
For containers a version controlled requirements.txt or Dockerfile is the way.
Realistically, Anaconda is a venerable solution from the truly god awful old days. The ecosystem is vastly improved.
>Due to lack of Python 3.11 support for packages that PyTorch depends on, including NumPy, SciPy, SymPy, Pillow and others on the Anaconda platform. We will not be releasing Conda binaries compiled with Python 3.11 for PyTorch Release 2.0. The Pip packages with Python 3.11 support will be released, hence if you intend to use PyTorch 2.0 with Python 3.11 please use our Pip packages.
It really sucks that anaconda always lags behind. I know the reasoning*, and I know it makes sense for what a lot of teams use it for... but on our side we are now looking more and more into dropping it since we are more of an R&D team. We already use containers for most of our pipelines, so just using pip might be viable.
*Though I guess Anaconda chewed more than it can handle w.r.t managing an entire Python universe, and keeping up to date. Conda-forge is already almost a requirement but using the official package (with pip, in this case) has its own benefits for very complex packages like pytorch.