News for hackers. Stuff that matters.
Re:Making pip installs a little less slow by Galanwe on Sunday May 29, 2022 @12:28AM (#31546389)
The size of python packages is becoming an increasingly worrying issue.

A lot of maintainers don't pay attention to excluding test cases and artifacts from their packages, leading to ridiculous package size growth.

Last I checked numpy was close to 100MB, a huge chunk of that being test case artifacts.

Django packages all translations for all languages to ever exist.

Pandas bundles pre-built dynamic libraries with debug symbols.

Etc.

Most of my "production" virtualenvs are close to a GB nowadays, which is insane.

Re:Making pip installs a little less slow by houzi on Sunday May 29, 2022 @07:38AM (#31548306)
This! Especially with pandas.

I did have a look at numpy and on my machine the tests did not bloat it as much as you made me believe. The `core/tests/` modules are 3MB and the `__pycache__` doubled it to 6MB. What do you refer to as "test case artifacts"? The modules, pycache, or both?

Also, wrt you statement on pandas; is it the debug symbols that account for the bloat, or the libs themselves?

Re:Making pip installs a little less slow by Galanwe on Sunday May 29, 2022 @10:22AM (#31549346)
I cannot seem to remember which version exactly that was. I remember there were some test input/output files that weighted multiple MBs.

Just installed a quick "data science" like virtualenv, here is the example things I talk about:

$ du -sh scipy 110M scipy $ find scipy -type f -name '.so' -exec strip -s {} \; $ du -sh scipy 76M scipy

$ du -sh django/contrib/admin/locale/ 5.4M django/contrib/admin/locale/

Re:Making pip installs a little less slow by tedmiston on Sunday May 29, 2022 @01:13AM (#31546595)
All of those cases sound like good reasons for optionals / extras.
Re:Making pip installs a little less slow by boxed on Sunday May 29, 2022 @09:43AM (#31549066)
I've had people report missing tests from my python packages as bugs. You can't please all the people all of the time.
Re:Making pip installs a little less slow by kzrdude on Sunday May 29, 2022 @05:10AM (#31547672)
Sounds like a bug should be filed towards numpy?
Re:Making pip installs a little less slow by kzrdude on Sunday May 29, 2022 @06:18AM (#31547983)
Apparently it was wontfixed last year, but maybe it could still be suggested to be an extra, not in the standard package? https://github.com/numpy/numpy/issues/19591
Re:Making pip installs a little less slow by LosWochosWeek on Sunday May 29, 2022 @07:23AM (#31548248)
That was implied in the thread and subsequently wonfixed.

> Sure, but it's straightforward to install those [0]. A separate test suite would not be.

0: Those = Other packages you would have to download to make use of the testing components, i.e. pytest

Re:Making pip installs a little less slow by kzrdude on Sunday May 29, 2022 @09:27AM (#31548966)
What about pip install 'numpy[tests]' i.e using the extras feature?