Re:Conda: A Package Management Disaster?
by macinjosh on Friday December 20, 2024 @09:17PM
(#42477003)
That poor package server getting 39 simultaneous pulls at the same time from one user.
Re:Conda: A Package Management Disaster?
by AnthonBerg on Saturday December 21, 2024 @05:35AM
(#42478813)
This is indeed something to consider!
Not as an excuse for bad behavior but rather to consider infrastructure and expectations:
The packages might be cached locally.
There might be many servers – a CDN and/or mirrors.
Each server might have connection limits.
(The machine downloading the packages miiiiiight be able to serve as a mirror for others.)
If these are true, then it’s altruistically self-interested for everyone that the downloader gets all the packages as quickly as possible to be able to get stuff done.
I don’t know if they are true. I’d hope that local caching, CDNs and mirrors as well as reasonable connection limits were a self-evident and obviously minimal requirement for package distribution in something as arguably nation-sized as Python.
And… just… everywhere, really.
It is entirely possible to use poetry to determine the precise set of packages to install and write a requirements.txt, and then shotgun install those packages in parallel. I used a stupidly simple fish shell for loop that ran every requirements line as a pip install with an “&” to background the job and a “wait” after the loop. (iirc) Could use xargs or parallel too.
This is possible at least. Maybe it breaks in some circumstances but I haven’t hit it.