How would you like modules to work?

It seems great to me that paths & source files are mostly irrelevant, you're free to re-organise without changing anything. And that `using Xyz` is always talking to the package manager. You can make sub-modules and say `using .Xyz`, but there's very little need to do so, and few packages do.

You can shoot yourself in the foot by including source twice, as you can by generating it with a macro, or simply copy-pasting wrong.

I mean, I'd like them to work like Python, Ruby and TypeScript, but you're right to say I can't describe why I want this.

Is there some guide I could read about structuring a large Julia project? It was pretty easy to intuit with Python, wherein I would put related files in a folder. But with Julia, everything is everywhere and I'm baffled.

Just reading what everyone does may work. Here's a tiny package:

https://github.com/ChrisRackauckas/EllipsisNotation.jl

I think that `/src/Name.jl` must have the main module, and `/test/runtests,jl` tests. And the package manager cares about `Project.toml`. But beyond this there are no real rules enforced, although there really seems to be one way to do things.

Here's a much bigger project, organised the same way. `include(file.jl)` literally copies in the text, and it's somewhat conventional to collect all imports & exports in the main file:

https://github.com/JuliaNLSolvers/Optim.jl/blob/master/src/O...

Still no sub-modules. No files included in mysterious locations. Methods being defined for functions from elsewhere are all qualified, like `Base.show(io::IO, t::OptimizationState) = ...`

> But with Julia, everything is everywhere and I'm baffled.

This is exactly it. Julia allows you to import and include anything, anywhere. You open a file and it doesn't say anything about where the dependencies are coming from and where this particular piece of code will go. Both of those are defined at the place where this file is included, which itself could be anywhere. It could be a different directory, different file, tucked away in a module. It could be in a dozen other files, or no files at all, and you can't tell from looking at just the source of the file.

In practice most packages:

1. Never use submodules (they don't add much) 2. Only use `include` within the main `src/MyPackage.jl` file

Languages like Python, Rust, C# or even Java have module systems that I find are more restrictive, but much easier to follow. You always have the pertinent information at hand. Each file containing code clearly tells you two crucial pieces of information:

1. Where the code fits in the greater picture

2. Where the dependencies of the code in a file come from

Python, whose module story is actually pretty poor, is still easier to follow than Julia, because it just matches the file/directory structure. You can reason about the hierarchy of a python library by just navigating the directories. In a normal python project, each file is one module and it's dependencies are clearly specified as imports.

Rust relies on the file system as well, and much better defined rules than Python. I find this great, because the file system is already hierarchical and we are used to the way it works. When I open a file in a Rust project, I know immediately where it fits in the hierarchy - because it is implied from the file system. Rust gives you a bit more flexibility in that you can define submodules in each file.

C# & Java qualify the namespaces fully in each file. While the file structure is not as clear anymore at the file system level, a single file contains all the information necessary to determine where the code fits in and where it's dependencies come from.

Now let's take Julia:

A single module will be often split across multiple files. Since they share a single namespace, the imports happen at the top level where the module is defined and includes all it's source files. When you open a source file, you have zero information where all the functions and data types are coming from (or where they are going for that matter).

I see the following pattern systematically emerge in Julia code:

- A function is defined in file A

- File A is `included` in file B, where it forms part of module X

- It is then imported into module Y in file C, but it is not actually used there

- As it is finally used in file D, which is `included` in module Y in file C itself

The problem is that there is no link from file A to file B, or module X for that matter. File A could be part of a dozen modules, or zero. Neither is there a link between the usage of the function in file D and where it is coming from. You actually have to find all the places where file D is included, and then check what flavor of the function does each location import. The relationships are established at some indeterminate level in the hierarchy.

Again, don't get me wrong, this is just a wart on an otherwise very pleasant language. I wouldn't be complaining if I weren't using it.

That was helpful, especially realizing that part of the problem (at least as you see it) is that the "linking" is unidirectional.

Function definitions/calls are also unidirectionally linked. You can see at the call site which function is called, but you can't see at the function definition the references. But unlike a function, which might be called from many places, it really should be the case that a file is `include`d exactly once.