News for hackers. Stuff that matters.
Re:Clean your codebase with basic information theory by Chris_Newton on Monday February 19, 2024 @06:16AM (#39428510)
I have actually started to think that, against conventional wisdom, everything should be a global variable. All the issues with global variables can be solved with better tracing tooling.

This is an interesting premise, and actually I think we have quite a lot of examples both of successfully applying the idea up to a point and of where it starts to break down in practice.

Modern distributed applications mostly have a back end with some kind of database and front end UIs that depend on that back end via an API. Those databases are often global data stores, accessible from anywhere in the back end implementation. A lot of work is done to design and manage them, probably modelling some real world system that our application is concerned with, and there are varying degrees of abstraction/isolation used to preserve that design intent.

If the data model is simple then this works OK, particularly if you have a SQL database that can enforce some basic constraints and relationships to make illegal states unrepresentable.

What we usually see as the data models and the actions that update them become more complicated is the introduction of some business logic layer. The rest of the system isn’t allowed free access to update the state any more; it’s required to go through some defined interface that provides specific actions that guarantee the state remains valid.

That’s the writing side. On the reading side, aside from security/privacy issues, we generally don’t have the same concerns with allowing free access to the whole database from anywhere. However, often we need some form of derived data that isn’t directly stored in the database itself but instead can be constructed from other state that is. So again we end up with some kind of abstraction/isolation layer between the rest of our system and the database.

In each of these cases, there is probably data that we’re working with that is not the state that ultimately persists within the database. So the question immediately arises, if we only have global data in our programs, where does all of this transient, intermediary data go? If we put it into our database as well then all the usual problems with concurrency and integrity immediately appear, so we are back to needing something that is local to our immediate logic and can’t conflict with any other logic or indeed any other instance of the same logic that happens to be running in some other context at the same time.

We see analogous issues in the front end UI code for those distributed systems. If there is a relatively simple model then maybe the front end can effectively just fetch/cache the state from the back end API. As things get more complicated, maybe you end up with a front end data store analogous to the back end database that becomes the central, authoritative store of your front end state. And again maybe this provides some defined interface for accepting valid updates to the state and/or for accessing derived data. And again the questions arise about where the intermediate data generated by all of that logic should go if we have only our global store to hold state, and the answer is likely to be some form of more local data.

On top of the persistent state and anything acting upon or derived from it, we also have other kinds of information we work with in front end code. Many UIs will have state that is used purely to control the user’s view into the inner world: the sorting criteria and current page of a table, the current position and zoom level over a map, the last item we’re currently showing in an infinitely scrolling list, look and feel settings like whether we’re using a dark mode theme. Some of this data might apply across the whole UI while other aspects might only apply to, say, a specific table, with each table needing its own instance of that “UI state” data. So again, if everything were global, that would mean we’d need to include every possible piece of UI state for the whole application in the global store.

This comment is already far too long so I’ll just quickly note that there are other recurring themes. One is how to synchronise “global” stores in a distributed system where you might have multiple front ends running with their own copies of the state, or perhaps multiple microservices on a back end that have duplication in their databases because everything is supposed to be independent and denormalised. A related issue is how to represent temporary clones of significant parts of the data during user interactions, like building up a transaction with several changes before atomically committing it or rolling it back (think dialog box on the UI side or an internally consistent batch of changes sent to the back end), or supporting an undo facility that needs to reconstruct a previous version of the persistent state one way or another.

I do believe there’s a lot more we have to learn about different types of state and transient data and how we can model those cleanly in our systems. There are certainly common patterns we touch on in a lot of different contexts. And I think both extremes of having too much data trapped too locally and having too much lifted to global storage have their own difficulties and probably there is some sort of structure in between that would be better than what we typically write today. But it’s not an easy problem or we’d all be solving it by now…

Re:Clean your codebase with basic information theory by valty on Monday February 19, 2024 @10:01AM (#39430463)
The comment was mainly related to in-memory variables within an application process...focusing on scoping/syntax...but the thinking was definitely inspired by the fact that most apps center around an external database without realizing its essentially a global variable.

In application code, when I talk of global vars, I mean that every function has access to all data...as opposed to access being abstracted and modularized into various services which are exposed via being passed through a chain of function args, or some kind of dependency injection system.

But this global variable could actually be an abstraction (a store) allowing data integrity checks on writes.

> there is probably data that we’re working with that is not the state that ultimately persists within the database

If you think about all your data in one big graph, this transient data still has a relation to the final persisted state. There is a data flow of transient values into the persisted data values. And separately, your intermediate data structures might also contain relations to your persisted data structures.

Most dev tools don't track these relationships, and you have this tangled ad-hoc mess where data is dumped from one structure to the next.

> “UI state” data...we’d need to include every possible piece of UI state for the whole application in the global store.

Yep, this is what you should do.

If I am sorting something on a webapp and I refresh the browser, I probably want to have see the same things as they were sorted. This might vary between use case, but adding the functionality should be easy to do if necessary. So therefore it is good practice to allow all local ui state to be persisted by default.

The ui state is being persisted anyway, inside a component or inside the HTML document. Somewhere in the heap this data is stored. And if we think about our one big graph again, this data is related to other things...its just that we lose these relations.

> A related issue is how to represent temporary clones of significant parts of the data

All rendered UI values should be in their own _ui_ models (view model is similar), separate from the source of truth models.

These ui models basically allow all rendered UI to be editable without immediately committing changes to the database. This allows for optimistic UI updates. They get notified of any incoming changes from the source of truth, and can decide what to do with them.

If you want to batch them up, you just create a Batch entity, and add a relation to these ui models. The main thing is to treat the ui models like any other models. Whether they are persisted or not should simply be flipping a flag in your code.

For UI, everything should be in one big graph. Code is data.

I find with modern programming, all of the popular programming languages, frameworks, libraries, databases, platforms, really get in the way of being able to do things simply.

Re:Clean your codebase with basic information theory by PeterisP on Monday February 19, 2024 @10:35AM (#39430883)
In this context when people say "global variable" it is effectively shorthand for "arbitrary accessible shared mutable state" and the associated risks of mutating something which other code expects to stay the same, and also often unintentionally coupling things without it being clear that they are coupled.

I'd argue that an external database is not necessarily such a thing - it can be and sometimes is, but in many (most?) applications there generally is a somewhat well-defined layer that manages that state mutation and enforces any constraints needed, isolating the "global variable" from the rest of the program, just as a singleton class with constrained setter functions could do for an OOP approach to a shared state that's arguably fixes most issues with "naked" global variables; and standard data flow analysis in specifications can make explicit any coupling between components that happens "through the DB".

Re:Clean your codebase with basic information theory by Chris_Newton on Tuesday February 20, 2024 @03:37PM (#39446571)
Again, I like a lot of the thinking behind this. I’m personally a big fan of having clear data models and data flows, and I think there is a lot of potential in ideas like reactivity that clearly define sources of truth and relationships in single places so derived state gets updated automatically when its underlying source(s) of truth change. I also believe realising the full potential of some of these ideas will, as you’ve mentioned yourself, need better tools than we currently have. However, we can already get pretty far with these ideas, and when they work well, it certainly does make for a nice, clear, easy-to-follow design compared to having state and relationships scattered all over the code base.

Where I suspect we might disagree is the idea that everything should fit into this model. Sometimes the data models and relationships get more complicated and the one-way flows implied by the basic idea of reactivity are no longer sufficient. Sometimes the shape of the data flow graph changes on the fly, because we need multiple instances of certain parts of the state. Then we need a way to distinguish between those instances and often some way to connect the repeated parts of the graph back to the wider graph so that we can make changes to the duplicated parts but still enforce relationships with other parts of the data model.

I have never so far seen a reactivity-based approach to modelling state that handles more demanding scenarios like these gracefully, nor a convincing theory for doing so without introducing more than just a general global data store and simple reactivity. This is where I believe there is still a lot of potential for finding better ways to model the data and then better tools to implement those models. The trick, as again you said yourself, is to find the essential aspects and make it simple to describe them without the accidental complexity from the tools becoming excessive.