Learning how to use a simple barebones venv is extremely easy, saves a ton of time both in the short and long run, and generalises better.
pip install virtualenv
cd your/project/location
which python
virtualenv -p result_of_which_python env
source env/bin/activate
pip install anything_you_like
> but I use a "base" folder where I have already downloaded all the packages I use by default [...etc...]
Your setup sounds outrageously complex, and I don't understand why you would do any of that.
cd your/project/location
python3 -m venv env
source env/bin/activate
pip install anything_you_like
py -3.6-32 -m venv env
[1] https://docs.python.org/3/using/windows.html#python-launcher...
- It would work the same way on Windows, Linux and macOS
- Most importantly (!), it is not a Python package manager. That is, if a package needs BLAS or MKL or HDF5 or whatever else, you won't be crossing fingers and hoping your system-installed version would work for all your venvs; instead, those binary libraries are properly managed per environment.
Pro tip: use mamba instead of conda to get a free 4x speed boost.
That's the generalisation part I mentioned. The distinction you're making is non-trival for a wee noobie. Better they get the ground work in and expand to where they need to go.
Then they're reading on some blog about protobuf, or hdf5, or arrow or whatever, and want to use it - but using either from Python needs installed C libraries. On linux/macos you'll need to dig into your package managers and then hope things are compatible, on Windows it's a complete pain. In conda, they can just 'conda install h5py' or whatever, and get proper hdf5 installed without having to figure out the nitty gritty details.
Well, already I can tell you're disconnected from what "most noobs" are actually like. Presumably you think "most noobs" are data scientists, which just isn't the case in my experience.
It is the case in my personal experience though. I don't think I'm disconnected from what 'noobs' are as I've taught a lot of them through my line of work.
Assume you're a noob and are exploring Python package universe, fire a new private tab and google 'most popular python packages'; in my case the first 3 links are:
- https://www.activestate.com/blog/top-10-must-have-python-pac...
- https://www.ubuntupit.com/best-python-libraries-and- packages-for-beginners/
- https://www.edureka.co/blog/python-libraries/
All of which list numpy, pandas and the rest e.g. tensorflow and friends.
Please let them actually use python before sending them into tooling hell.
If you're just writing "a=b+c"-level Python scripts, or scrambling together tiny flask/django apps, or whatever else, you probably don't care indeed.
If you're doing/learning any kind of data-science-related stuff, that requires tons of C extensions and libraries. And many 'noobs' learn Python in order to use pandas/numpy/torch/whatever else is hot these days.
Of all the times I have tried conda only once was it as easy as following the guide they post. The other times it was completely broken. I also greatly dislikes that it pollutes your default environment. Imagine if every piece of software would do that.
But isn't that similar to what virtualenv does? The differences are that with virtualenv, folders are created elsewhere.
Besides that, friction so far hasn't enough for me to automate this even more (as in, creating a script that would do the folder creation) but I guess I'll do eventually.
Also, this way I have a set of python packages already installed.
And it was fun learning how to command pip / python to do things as I wanted them. That I guess answers this:
> I don't understand why you would do any of that
P.S.:
virtualenv -p $(which python) env
Isn't this what virtualenv was created to get away from?
If several of your projects rely on one of the "base" packages, upgrading it (which you might want for one project) could break the other projects.
Also, having implicit dependencies on packages might lead to problems with deployment (or for other developers) because they don't know that this package is required (and you might not have realized either, since it's always available on your machine).
(I might have misunderstood your solution, though. I've never used Docker.)
This other comment [1] clarifies my whole set up, and sheds a lot more light on how it works
The friction here sounds like a tire fire. I think you are just not aware at the moment of how much less convoluted this can be. You've become used to this complexity.
OK. I understand where are you coming from, and I feel your pain, being a developer myself and have been way too many times that I want to admit on the "But, why?!" position.
You sound really distressed by this, and at the same time honestly curious (And baffled can probably shoved in there) so 'll explain things a little more.
I think all this starts with the fact that I explained myself rather poorly. I _do_ have different envs for different codebases. My set up is like this:
Everything python related is in a folder where I store my "dockerized" projects:
~/projects/docker/python/
UN_PIP_TARGET=/work/packages
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/work/packages/bin
UN_PYTHONPATH=/usr/local/lib/python3.7/site-packages:/work/packages
PYTHONPATH=/work/packages
PYTHONUSERBASE=/work/packages
PIP_USER=yes
then I have a "base" project folder, that I use as a starting point for _most_ of my projects:
base
├── packages
├── notebooks
Whenever I want to start a new project, the steps I need to do is:
python.sh env_name
Once the folder is there, it just starts a container in interactive mode with the (environment) base folder always shared as /work
In case I need some files to work there, I just copy them.
Again, the _only_ issue so far - that annoys me -, is that if I create files when inside the container they are owned by root. Eventually I'll grow tired of this and will fix it, but not there yet.
for me starting a certain env is just one line of code. Obviously you could do the same with your method.
Said that I didn't have the script, for me creating a new env goes like this:
cp -r ./base new_env
docker run -it -rm --name new_env_container -v $(pwd)/new_env:/work -w="/work" python:3.7 /bin/bash
rm -r new_env
This way, the container is removed every time you exit it, but it has a name so you can log into another console would you need it. the only issue with this is that it's a barebones OS, so if you need some program for it you have to install it (And that is lost of you don't keep the container: in my todo list there's a change for python.sh where it would be possible to persist the containers, and run them from the one already existing if found. The downside to this is that every container you persist is several hundred mb... But space is cheap, right?). BTW, I do have a container I persist for use with youtube-dl as it depends heavily on ffmpeg.
From my point of view, that works the same a venv. Your mileage might vary.
Caveat: python:3.7 is not the default name of the python image; I renamed it because the default was too long. I used to do it this way until I got tired of writing the same long command.
virtualenv envbut otherwise, i agree entirely
In my case, and contrary to what one of the other commentators said, I'm using docker.
It took a while to get it ironed out, but I use a "base" folder where I have already downloaded all the packages I use by default (Top of my mind are pandas, numpy and youtube-dl ;) )
I have all the related configuration in an env file that tells pip to save the packages in the base folder so they don't disappear when I shut down the container - that I always run with --rm so they get removed when dead - and for creating a new env I just have to copy that folder. As I use a base folder for all of this I don't need to remember the names of the envs, as just need to list folders.
Only downside to this - because I'm lazy and it still hasn't bothered me enough to fix it - is that new files & folders are owned by root.
I use just a base image with python, and for different versions that work with this - supposedly - I just have to download a different image.