Those tools are essential and they basically rely on locality (context).
> Tight integration of the environment with the storage format brings some of the nicer features of database normalization to source code. Redundancy is eliminated by giving each definition a unique identity, and storing the name of variables and operators in exactly one place.
---
Why do we need modules at all?
This is a brain-dump-stream-of-consciousness-thing. I've been thinking about this for a while.
I'm proposing a slightly different way of programming here The basic idea is
- do away with modules
- all functions have unique distinct names
- all functions have (lots of) meta data
- all functions go into a global (searchable) Key-value database
- we need letrec
- contribution to open source can be as simple as contributing a single function
- there are no "open source projects" - only "the open source Key-Value database of all functions"
- Content is peer reviewed
... ---
Whole thread's worth a read.
[0]: https://erlang.org/pipermail/erlang-questions/2011-May/05876...
IMHO the key is visualization of everything. Data dependencies, runtime control flow, and a centralized graph-based data store and schema.
Intentional Programming is an interesting read as someone has already mentioned...from the guy who brought us `strHungarianNotation`. Storing code in a database but retaining the joy of the plain text cut/copy/paste experience is the key challenge, as well as all the unix file goodness.
Its quite fun to talk to ChatGPT about these topics and just question everything and delve back in the history of programming.
Unison plays with these ideas (I tried it, it's taking things in the right direction, though I still can't figure out how to write anything more complex than sorting numbers in the REPL with it; the examples are too Haskelly, IMHO.) Smalltalk language is, I believe, the original - built around the assumption that code is in the database, and coming with a built-in IDE for this. Glamorous Toolkit is trying to push this further, to give programmers better ability to create ad-hoc problem-specific views into their programs.
I've seen a few other articles written about this over the years, but I don't have any link handy.
What current languages are bad about is expressing architecture, and the problem of having one way to structure our models (domain models) vs. the actions/transformations that run on them (flow of execution).
I strongly disagree on the global variable side though...
My thinking is that software has been terrible (over-complex) for such a long time, so its time to start questioning our most dogmatic principles, such as "global variables are bad".
Imagine you can instantly see all the dependencies to/from every global variable whenever you select it. This mitigates most of the traditional complaints.
I would argue that adequate tooling that allows for this would dramatically simplify all development. It's the only thing that matters and its so absent from every development platform/language/workflow.
If we could only see what was going on in our programs, we would see the complexity, and we could avoid it.
Another related bit of dogma is _static scoping_. Why does a function have to explicitly state all its arguments? Why aren't we allowed to access variables from anywhere higher up in a call stack?
What you realize is that all of these rules are so you can look at plain text code and (kind of) see what is going on. This is a holdover from low-powered computers without GUIs like most of programming. Even if an argument is explicit, if its passed down via 10 layers, you still have to go look.
E.g. dynamic vs lexical scoping. Dynamic scoping used to be more popular and you can still use it in some languages like elisp. In some situations it's a natural fit for the problem, but I think in most cases lexical scoping is simply easier to use.
> easier to use
With plain text editors for sure. You really need a mandatory re-imagined IDE to make it work.
You need to be able to see exactly where the variable is coming from...which I think would be a good feature anyway.
And for this you really need a live programming environment...which I think would be good too...but they are very rare. Everyone is obsessed with static typing these days, but runtime value tracing is more useful imho.
Having code be tree like or at least a DAG is how you resolve what is probably most easily visualized as dependency hell.
This is why many modern patterns try and steer people into having their domain logic centered and compromised of idempotent functions.
And while they didn't know why back then, it is why accounting has used immutable events for over 500 years.
The notion of free variable is another lens to think about it.
Globals are a tool with tradeoffs.
Even with excellent introspection and debugging tools, it's hard to prove anything about the state of a mutable global variable (since it's hard to reason about the effects of many interleaved instructions), so it's hard to prove anything about your program that depends on that state (like whether the program is correct enough), and code accessing that global must be more complicated to account for the fact that it doesn't know much about it.
Suppose you have something that's effectively a global anyway (logging configuration), isn't mutable (math constants), or for some reason you can actually only have one of (memory page table). Sure, you probably gain a lot by making the fact that it's global explicit instead of splattering that information across a deep, uncommented call chain.
Could other use cases exist? Sure. Just be aware that there's a cost.
> dynamic scoping not bad
It's not just a matter of visibility (though I agree with something I think you believe -- that more visual tools and more introspectability are extremely helpful in programming). No matter whether you use text or interactive holograms to view the program, dynamic scoping moves things you know at compile-time to things you know only at run-time. It makes your programs slower, it makes them eat more RAM, and it greatly complicates the implementation of any feature with heap-spilled call stacks (like general iterators or coroutines).
Implementation details vary, but dynamic scoping also greatly increases the rate of shadowing bugs -- stomping on loop iterators, mistyping a variable and accidentally using an outer scope instead of an inner scope, deleting a variable in an outer scope without doing a full text search across your whole codebase to ensure a wonky configuration won't accidentally turn that into a missing variable error at runtime 1% of the time, ....
Modern effect systems actually look a lot like a constrained form of dynamic scoping, and some people seem to like those. Dynamic scoping isn't "bad"; it just has costs, and you want to make sure you buy something meaningful.
This is an interesting premise, and actually I think we have quite a lot of examples both of successfully applying the idea up to a point and of where it starts to break down in practice.
Modern distributed applications mostly have a back end with some kind of database and front end UIs that depend on that back end via an API. Those databases are often global data stores, accessible from anywhere in the back end implementation. A lot of work is done to design and manage them, probably modelling some real world system that our application is concerned with, and there are varying degrees of abstraction/isolation used to preserve that design intent.
If the data model is simple then this works OK, particularly if you have a SQL database that can enforce some basic constraints and relationships to make illegal states unrepresentable.
What we usually see as the data models and the actions that update them become more complicated is the introduction of some business logic layer. The rest of the system isn’t allowed free access to update the state any more; it’s required to go through some defined interface that provides specific actions that guarantee the state remains valid.
That’s the writing side. On the reading side, aside from security/privacy issues, we generally don’t have the same concerns with allowing free access to the whole database from anywhere. However, often we need some form of derived data that isn’t directly stored in the database itself but instead can be constructed from other state that is. So again we end up with some kind of abstraction/isolation layer between the rest of our system and the database.
In each of these cases, there is probably data that we’re working with that is not the state that ultimately persists within the database. So the question immediately arises, if we only have global data in our programs, where does all of this transient, intermediary data go? If we put it into our database as well then all the usual problems with concurrency and integrity immediately appear, so we are back to needing something that is local to our immediate logic and can’t conflict with any other logic or indeed any other instance of the same logic that happens to be running in some other context at the same time.
We see analogous issues in the front end UI code for those distributed systems. If there is a relatively simple model then maybe the front end can effectively just fetch/cache the state from the back end API. As things get more complicated, maybe you end up with a front end data store analogous to the back end database that becomes the central, authoritative store of your front end state. And again maybe this provides some defined interface for accepting valid updates to the state and/or for accessing derived data. And again the questions arise about where the intermediate data generated by all of that logic should go if we have only our global store to hold state, and the answer is likely to be some form of more local data.
On top of the persistent state and anything acting upon or derived from it, we also have other kinds of information we work with in front end code. Many UIs will have state that is used purely to control the user’s view into the inner world: the sorting criteria and current page of a table, the current position and zoom level over a map, the last item we’re currently showing in an infinitely scrolling list, look and feel settings like whether we’re using a dark mode theme. Some of this data might apply across the whole UI while other aspects might only apply to, say, a specific table, with each table needing its own instance of that “UI state” data. So again, if everything were global, that would mean we’d need to include every possible piece of UI state for the whole application in the global store.
This comment is already far too long so I’ll just quickly note that there are other recurring themes. One is how to synchronise “global” stores in a distributed system where you might have multiple front ends running with their own copies of the state, or perhaps multiple microservices on a back end that have duplication in their databases because everything is supposed to be independent and denormalised. A related issue is how to represent temporary clones of significant parts of the data during user interactions, like building up a transaction with several changes before atomically committing it or rolling it back (think dialog box on the UI side or an internally consistent batch of changes sent to the back end), or supporting an undo facility that needs to reconstruct a previous version of the persistent state one way or another.
I do believe there’s a lot more we have to learn about different types of state and transient data and how we can model those cleanly in our systems. There are certainly common patterns we touch on in a lot of different contexts. And I think both extremes of having too much data trapped too locally and having too much lifted to global storage have their own difficulties and probably there is some sort of structure in between that would be better than what we typically write today. But it’s not an easy problem or we’d all be solving it by now…
In application code, when I talk of global vars, I mean that every function has access to all data...as opposed to access being abstracted and modularized into various services which are exposed via being passed through a chain of function args, or some kind of dependency injection system.
But this global variable could actually be an abstraction (a store) allowing data integrity checks on writes.
> there is probably data that we’re working with that is not the state that ultimately persists within the database
If you think about all your data in one big graph, this transient data still has a relation to the final persisted state. There is a data flow of transient values into the persisted data values. And separately, your intermediate data structures might also contain relations to your persisted data structures.
Most dev tools don't track these relationships, and you have this tangled ad-hoc mess where data is dumped from one structure to the next.
> “UI state” data...we’d need to include every possible piece of UI state for the whole application in the global store.
Yep, this is what you should do.
If I am sorting something on a webapp and I refresh the browser, I probably want to have see the same things as they were sorted. This might vary between use case, but adding the functionality should be easy to do if necessary. So therefore it is good practice to allow all local ui state to be persisted by default.
The ui state is being persisted anyway, inside a component or inside the HTML document. Somewhere in the heap this data is stored. And if we think about our one big graph again, this data is related to other things...its just that we lose these relations.
> A related issue is how to represent temporary clones of significant parts of the data
All rendered UI values should be in their own _ui_ models (view model is similar), separate from the source of truth models.
These ui models basically allow all rendered UI to be editable without immediately committing changes to the database. This allows for optimistic UI updates. They get notified of any incoming changes from the source of truth, and can decide what to do with them.
If you want to batch them up, you just create a Batch entity, and add a relation to these ui models. The main thing is to treat the ui models like any other models. Whether they are persisted or not should simply be flipping a flag in your code.
For UI, everything should be in one big graph. Code is data.
I find with modern programming, all of the popular programming languages, frameworks, libraries, databases, platforms, really get in the way of being able to do things simply.
I'd argue that an external database is not necessarily such a thing - it can be and sometimes is, but in many (most?) applications there generally is a somewhat well-defined layer that manages that state mutation and enforces any constraints needed, isolating the "global variable" from the rest of the program, just as a singleton class with constrained setter functions could do for an OOP approach to a shared state that's arguably fixes most issues with "naked" global variables; and standard data flow analysis in specifications can make explicit any coupling between components that happens "through the DB".
Where I suspect we might disagree is the idea that everything should fit into this model. Sometimes the data models and relationships get more complicated and the one-way flows implied by the basic idea of reactivity are no longer sufficient. Sometimes the shape of the data flow graph changes on the fly, because we need multiple instances of certain parts of the state. Then we need a way to distinguish between those instances and often some way to connect the repeated parts of the graph back to the wider graph so that we can make changes to the duplicated parts but still enforce relationships with other parts of the data model.
I have never so far seen a reactivity-based approach to modelling state that handles more demanding scenarios like these gracefully, nor a convincing theory for doing so without introducing more than just a general global data store and simple reactivity. This is where I believe there is still a lot of potential for finding better ways to model the data and then better tools to implement those models. The trick, as again you said yourself, is to find the essential aspects and make it simple to describe them without the accidental complexity from the tools becoming excessive.
You may be interested in the programming language I've been working on :)
I didn't find out about Unison until much later :) Cool project though!
I would argue this problem is solved in most current languages with strict types.
Stop making all the things strings or abstract base classes.
Easy example I've worked on recently. An IPv4 address is an IPv4address in code. I don't care if it is just represented as an uint32 or string in Memory, in your code it should be an IPv4 address and if a function expects an IPv4 address and you pass it a string that is a compilation error.
the indexing step is basically building this db in the background, it's just kept out of view / hidden unless you're building ide plugins or whatever.
Value tracing is at runtime. JetBrains cannot trace how values flow through your code.
To do this, you need to instrument all your code, and track all the transformations that occur for each value. It's really difficult to do if the language is not designed for it and there are a lot of performance implications.
If your code is written in a functional paradigm it becomes much easier to trace...such as with rxjs.
TL;DR: locality is likely one of the most important concepts in all of knowledge work. Of course we're a little obsessed with it.
This is the real problem that needs solving.
> code that is read together should be written together
Code is a database of functions. This approach is like trying to design a database in denormalized form.
No matter how efficient we are overall, what we can access the fastest is fundamentally limited, because what we can keep closest is limited. If you want to access information in less than 3 nanoseconds, a copy of that information must be stored less than a meter away.
More prosaically, the reason our screens offer only a small window, is because our eyes only offer a small window. The problem would not really be solved even with infinite resolution VR googles.
> Code is a database of functions. This approach is like trying to design a database in denormalized form.
I believe I disagree for the most part.
I think this pursuit of "locality" is what actually causes more complexity. And I think its mainly around our obsession with representing our code as text files in folder hierarchies.
> coarsely structure codebases around CPU timelines and dataflow
This is why I would prefer code to be in a database, instead of files and folders, so that structure doesn't matter, and the tree view UI can be organized based on runtime code paths, and data flow - via value tracing.
> don’t pollute your namespace – use blocks to restrict variables/functions to the smallest possible scope
Everyone likes to be all modular and develop in tiny little pieces that they assemble together. Relying on modularization means that when stuff changes upstream in the call stack, we just hack around these changes adding some conditionals to handle these changes instead of resorting to larger refactors. People like this because things can keep moving instead of everything breaking.
Instead, what we need to do is make it easier to trace all the data dependencies in our programs so that when we make a change to anything, we can instantly see what depends on it and needs updating.
I have actually started to think that, against conventional wisdom, everything should be a global variable. All the issues with global variables can be solved with better tracing tooling.
Instead we end up with all these little mini-databases spread all over our code, when what we should have is one central one from which we can clearly see all the data dependencies.
> group related concepts together
Instead, we should query a database of code as needed...just like we do with our normalized data.