Making pip installs a little less slow

Re:Making pip installs a little less slow by edpenz on Sunday May 29, 2022 @01:40AM (#31546722)

The most efficient workflow I've found for pip so far is something along the lines of:

  pip-compile # resolve and pin the dependency tree
  grep '==' requirements-pinned.txt | xargs -n1 pip wheel  # build every wheel in parallel
  pip install *.whl  # install the pre-built wheels

It may not avoid all the slowness, but when nicely integrated into a build/CI system it can avoid suffering it more than once.

[ Parent | on HN ]

Re:Making pip installs a little less slow by BiteCode_dev on Sunday May 29, 2022 @04:40AM (#31547528)

It's interesting if you got a lot of wheels to build locally, but this is very specific: most wheels are pre-built for you, that kinda the point.

It still happens from time to time, but if somebody else read this comment, you want to check how many wheels you build before doing so.

Also, you may want to cache those build once and for all.

[ Parent | on HN ]

Re:Making pip installs a little less slow by raffraffraff on Sunday May 29, 2022 @07:22AM (#31548242)

Depends on your platform. If you're using ARM (like AWS Graviton, and possibly Apple M1) you might find that many wheels are not available. That might be different now, my information is 2 years old, but I remember 'pip install numpy' taking ages. Every time someone pushed a change we'd have a 15 minute job clogging the pipes. Luckily I had already created an artifact repo (AWS CodeArtifact) for our own internalpackages, so I added a step to our build process to push locally built wheels, so a painfully slow build would only happen once.

[ Parent | on HN ]