Namespaces are great but Python’s module system being coupled to file system structure kinda sucks. You either have overly-granular modules with manageable files, or you end up with multi-thousand line files that present a reasonable module organization. Or the totally unreasonable granular module structure for reasonable files with a facade set of modules that just reexport things.

You can split your big module into modules inside a sub-package and import stuff in the __init__.py file from the submodules to present a nice interface to the library user.

eg instead of:

    namespace.py
        class C1:
        class C2:

do

    namespace/__init__.py
        from .c1 import C1
        from .c2 import C2

    namespace/c1.py
        class C1:

    namespace/c2.py
        class C2:

What I don't like about this approach is how I'm supposed to refer to something by namespace.C1 but its repr is namespace.c1.C1.

I would prefer if the repr reported its primary public API name, not its internal implementation location.

That’s what I described as the facade module approach.

What's wrong with facade modules? That seems to be a pretty common pattern, and it allows you to decouple your filesystem structure from module structure while still giving someone reading the source a breadcrumb to follow.

Quite apart from implementation challenges (eg, needing to parse every file in package upfront to know what's in there), how else would you see this working? Are there other interpreted languages that manage this indirection in a more elegant way than having some central file that supplies the mapping information?

There’s nothing wrong with it, per se, but it is more to maintain, and it’s a confusing indirection to downstream users that need to pry into your code to debug something.

Ruby has its own problems, but I like that the modules are independent from the file that contains them. You can “reopen” a module and declare new classes or constants or whatever. You have freedom (and responsibility) to organize your files in a way that maps to module namespaces. The drawback is the `require “my_file”` doesn’t give you any hint about what you’re importing.

The module system linked to file structure worked for Perl pretty well. Perl had this crazy idea that different authors should publish packages/modules with the same namespace, e.g.

  IO.pm                      <-- author: Larry Wall
  IO/File.pm                 <-- author: Me
  IO/Socket.pm               <-- author: You
  IO/Socket/INET.pm          <-- author: Larry Wall
  IO/Socket/INET/Daemon.pm   <-- author: Me

Simpler and encourages code reuse. Of course there is duplicate code in CPAN, but you really have to go out of your way to avoid an existing module that can do what you need.

You can organize your packages however you want if you write and register a new module loader.

It violates rule 5 "Flat is better than nested".

NumPy also imports all of its subpackages, allowing:

  >>> import numpy
  >>> P = numpy.polynomial.polynomial.Polynomial

This results in a high startup cost if you just one one NumPy feature.

I've always thought this ran counter to "Explicit is better than implicit".

The NumPy developers think "practicality beats purity" is more important, and their main use-case is long-running programs where startup costs are slow, which I interpret as meaning their special cases is special enough to break the rules.

Isn't Numpy about making things go fast?

People who expect things to go fast want it all cached up front, for very obvious reasons.

Seems straightforward.

"Go fast" has multiple dimensions.

For one example, I had a command-line tool which needed to compute something related to hypergeometric distribution. (It's been a few years; I forget the details.)

This available in scipy, which need numpy. Most of my program's run-time was spent importing numpy. (The following timings are best-of-3.)

  % time python -c pass
  0.027u 0.011s 0:00.04 75.0% 0+0k 0+0io 0pf+0w
  % time python -c "import numpy"
  0.212u 0.059s 0:00.17 152.9% 0+0k 0+0io 14pf+0w
  % time python -c "import scipy"
  0.252u 0.077s 0:00.32 100.0% 0+0k 0+0io 14pf+0w

While 0.2 seconds doesn't seem like much to people used to spending hours developing a notebook, or running some large matrix computation, I could make my program 8x faster by writing the dozen or so lines I needed to evaluate that function myself.

Numpy is not about making short-lived programs fast.

In any case, this specific choice of importing all submodules is not to cache things up "for obvious [performance] reasons", because there is no machine performance improvements.

Instead, it's to make the API easier/faster to use. When you're in a notebook and you need numpy.foo.bar() you can just use it, and not have to go up and use an import statement first.

Yeah exactly.

However, what I'd say is that a short launch from rest is an even more obscure use case than the one numpy supports.

numpy has made the correct compromise IMO, optimizing for long lived programs and notebooks.

While my observation is that most other Python packages don't import all subpackages.

"import urllib" does not also import urllib.parse, so you can't do:

  import urllib
  urllib.parse.quote("A&W")

but instead must explicitly import urllib.parse.

scikit-learn supports essentially the same use cases as NumPy and it doesn't import all its subpackages:

  >>> import sklearn
  >>> sklearn.linear_model
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  AttributeError: module 'sklearn' has no attribute 'linear_model'
  >>> import sklearn.linear_model
  >>> sklearn.linear_model
  <module 'sklearn.linear_model' from '[...]sklearn/linear_model/__init__.py'>

Thus leading to my earlier comment:

> The NumPy developers think "practicality beats purity" is more important, and their main use-case is long-running programs where startup costs are slow, which I interpret as meaning their special cases is special enough to break the rules.

If you use a single namespace it becomes Monomial.