News for hackers. Stuff that matters.
Re:What to do about GPU packages on PyPI? by dataflow on Thursday May 20, 2021 @07:27PM (#27228734)
Bear in mind this isn't just end-users installing on their machines, it also includes continuous integration scripts that run quite frequently.
Re:What to do about GPU packages on PyPI? by viraptor on Thursday May 20, 2021 @08:26PM (#27229150)
> includes continuous integration scripts that run quite frequently

This use case is something I believe they could charge for if they need to cover infra costs. Same as Docker hub started doing - if someone fails to cache properly in their CI and wants to redownload things from the internet, they should pay for that.

Re:What to do about GPU packages on PyPI? by westurner on Thursday May 20, 2021 @07:58PM (#27228963)
[Huge GPU] packages can be cached locally: persist ~/.cache/pip between builds with e.g. Docker, run a PyPI caching proxy,

"[Discussions on Python.org] [Packaging] Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c...

> Continuous Integration automated build and testing services can help reduce the costs of hosting PyPI by running local mirrors and advising clients in regards to how to efficiently re-build software hundreds or thousands of times a month without re-downloading everything from PyPI every time.

[...]

> Request from and advisory for CI Services and CI Implementors:

> Dear CI Service,

> - Please consider running local package mirrors and enabling use of local package mirrors by default for clients’ CI builds.

> - Please advise clients regarding more efficient containerized software build and test strategies.

> Running local package mirrors will save PyPI (the Python Package Index, a service maintained by PyPA, a group within the non-profit Python Software Foundation) generously donated resources. (At present (March 2020), PyPI costs ~ $800,000 USD a month to operate; even with generously donated resources).

Looks like the current figure is significantly higher than $800K/mo for science.

How to persist ~/.cache/pip between builds with e.g. Docker in order to minimize unnecessary GPU package re-downloads:

  RUN --mount=type=cache,target=/root/.cache/pip

  RUN --mount=type=cache,target=/home/appuser/.cache/pip
Re:What to do about GPU packages on PyPI? by remram on Thursday May 20, 2021 @11:14PM (#27230086)
From an open-source developer's perspective, whether we hit PyPI (a third-party free service) or the cache provided by the CI service (a third-party free service) doesn't seem very different.
Re:What to do about GPU packages on PyPI? by captn3m0 on Friday May 21, 2021 @04:02AM (#27231565)
A 7TB local cache within the CI service is much cheaper to host, since it doesn't have CDN concerns, and bandwidth is much cheaper "within the infra".
Re:What to do about GPU packages on PyPI? by kortex on Friday May 21, 2021 @07:59AM (#27232925)
Wow, that RUN trick is exactly what I've been looking for! I've spent hours and hours in Docker documentation and hadn't seen that functionality.

Looks like it might be buildkit-specific?

Re:What to do about GPU packages on PyPI? by westurner on Friday May 21, 2021 @04:13PM (#27239300)
AFAIU `RUN --mount=type=cache` is specific to moby (Docker Engine) with `DOCKER_BUILDKIT=1`, though buildah does support a build-time volume mount option and moby easily could as well:

"build time only -v option" https://github.com/moby/moby/issues/14080

"Build images with BuildKit" https://docs.docker.com/develop/develop-images/build_enhance...