Every bit of state creates potentially exponentially more possible entity states. So therefore limiting potential changes in state limits the amount of working memory necessary to understand the system. Its starting with "can't" and then building a "can" when necessary, which is a lot better on memory, comprehension and feeling safe/secure to make changes then starting with a collection of 10^n "can"'s and adding in "can't"'s.
But that's immaterial to your main point, which is that adding state into the mix of things makes things hard. Which I agree with, but again to steelman the point, I could turn around and say that values allow for exponentially more possible values as well! When I see a map passed into a Clojure function I have no idea what could be in that map!
I think the main objection here which you are alluding to is one of "global" vs "local" reasoning. With a value I just need to worry about the body of my function, whereas with (global) state I need to worry about every function everywhere! But what if that's just a problem with our tools rather than an intrinsic issue? What if I had a tool that could automatically present all the mutable state of your system that is publicly accessible as a single screen and automatically link to different procedures that link to different parts of it? At that point I don't see much of a difference between state strewn everywhere and nice orderly values plumbed everywhere. In fact maybe it's nicer to have that implicit state strewn everywhere instead of having to carry around values which are irrelevant for the bulk of a function body and only relevant for a single part of a subfunction. What if it's all just a matter of not having the right IDE?
Working memory is definitely a hard limitation and universal enough among humans, but it's not clear to me it's a specific enough concern to convincingly justify certain programming language features which may just be crutches for inadequate visualizations or different educational backgrounds.
The world needs this. I think Pernosco has a workable technical foundation, but the GUI is a debugger and I need a code exploration tool to "find my way" in big unfamiliar codebases. Encouraging developers to pick up and hack around in others' codebases is the only way to get enough eyeballs to make all bugs shallow.
> maybe it's nicer to have that implicit state strewn everywhere instead of having to carry around values which are irrelevant for the bulk of a function body and only relevant for a single part of a subfunction.
I think global state (which is unusually bad) or shared mutable state (which is omnipresent outside of Rust) is a mental overhead (more things to keep in mind). I don't think tooling can eliminate the overhead of worrying about moving parts, only make it faster to look up (and hopefully document) what touches each bit of state.
> In fact maybe it's nicer to have that implicit state strewn everywhere instead of having to carry around values which are irrelevant for the bulk of a function body and only relevant for a single part of a subfunction.
I would call this an anti-pattern in FP. It's often a symptom of trying to replicate more imperative styles like OOP in a pure language. Threading mostly-irrelevant state through a bunch of different functions is a sign that your program is under-abstracted. If you think of all the function calls in your functional application as a tree, state should stay as close to the root of the tree as possible, kept in nodes it's relevant to, and the children and especially leaves of these nodes should be decoupled from it to the greatest extent possible.
The problem is that often you do want fairly complex state in the leaves of the tree, but want very little of it in anything else. Web browsers are a classic example of this. Pure FP solutions such as Elm that completely eschew the idea of local mutable state require a lot more ceremony to implement something like a form (the classic thorn for Elm users). By forcibly moving up the state to the root, you sometimes end up needing to pull some fairly severe contortions.
E.g. the usual answer to move the state back up to the root in the land of statically-typed, pure FP is to express it in a return type (e.g. a reader or state monad, culminating in the famous ReaderT handler strategy in Haskell) or in the limit bolt on an effect system instead. The usual answer in impure FP is to accept some amount of mutable state and just rely on programmers not to "overdo" it.
But from a certain point of view, writing an elaborate effect system whose very elaborateness might cause performance issues and inscrutable error messages sounds suspiciously like trying to work around a problem in visualization with an over-engineered code solution. And from another perspective it feels a bit like a trick. If some function has a lot of state, then I would hope by opening up the definition of the function I'd see how it all works, but with an effect system all of a sudden I've split things up into an interpreter that actually performs the mutation and an interface that merely marks what mutation is to be done. It feels like I've strewn logic around in even more places than if I just had direct stateful, mutable calls there!
I’m also very pragmatic and to the example of a web browser I would say, most applications are not web browsers. The overwhelming majority aren’t, in fact. I’ve chosen at this point in my career to mostly focus on enterprise software development, which I believe was Rich’s original field as well, and I’ve seen an enormous number of solutions with too much state cast about everywhere that benefit massively from centralizing the state high in the tree and really thinking through the data model carefully. So I stand by the principle I advocated originally, but it’s not universally applicable. It’s my belief that one of the core virtues of software development is knowing when to apply which principles.
I should've clarified. I meant developing a web page to run on a web browser, hence the form example.
In the case of Clojure, the map that you pass to a function is a value. It is guaranteed not to change underneath you and it can be freely shared with anybody.
> concurrency plus parallelism into the mix
The hard part of concurrency is writing or writing+reading, not just reading, so an immutable map isn't going to solve everything. Instead the hope is that you confine the mutability to one place with various transactional guarantees (in Clojure's case, this is usually atoms) and then everywhere else you don't have to worry about it.
But then again why couldn't the same analysis be performed on mutable state? How are we sure this isn't just a tooling issue? If we knew exactly what parts of mutable state were being touched by what we could identify what critical sections needed various guards.
Taking my hat off and going back closer to my own views, I actually think Clojure's combo of maps+atoms are an arguable case where Clojure has in fact complected things together in a way that e.g. STM doesn't (and Clojure's implementation and use of STM has its own problems). Namely it's complected committing a transaction with modifying an element in a transaction.
To illustrate the problem, right now Clojure atoms basically give up parallelism entirely. If you have a map in an atom with two threads modifying different keys, then those threads have to come one after another. It's actually kind of a waste of resources compared to the single thread case because work done in one thread will be thrown away and retried if the other thread wins.
So if you want true parallelism when modifying different keys you can use a ConcurrentHashmap. But that then gives up atomic updates of multiple keys at once! (Or you can have nested atoms but that has its own problems and doesn't solve the inter-key atomicity issue).
It looks like an all or nothing proposition where you either get non-parallel but fully atomic map updates or parallel per-key updates but nothing in-between. These kinds of false dilemmas are a classic symptom of complection.
The way other languages with an STM system deal with this is to build concurrent maps out of STMs refs. That way you get exactly the amount of parallelism you can relative to the amount of atomicity you need. If you have a transaction that touches two keys at once then both of those keys are atomically updated together and those two keys form one unit of parallelism. If you have a transaction that only touches one key then you have per-key parallelism. If you have a transaction that touches all the keys at once then you just collapse to the normal case of a map inside an atom.
As far as I can tell the reason Clojure doesn't do this (but other languages have) is that its STM API is a bit clunky and missing some interesting combinators.
All this is to say that maybe indeed simplicity and ease aren't all that different if from one perspective atoms are simple and from another merely easy.
I'm not going to delve into STM because that can be a whole book worth of discussion :). It's a fascinating universe, I've spent many hours (weeks, months?) exploring it, and I don't consider myself even close to an expert.
You are absolutely correct about the trade-off about atoms in Clojure.
Practically speaking, to start seeing retries you'd have to have a big number of updates going on at the same time. You can push a huge number of updates through a single thread. If you do have the need to do big throughput, you can explore not-so-idiomatic options like atoms-in-atoms, like you said.
IMO, the biggest unique benefit of combining atoms with immutable persistent data structures, comes from the fact that you can get unlimited number of consistent readers virtually for free. Any thread can look at (aka, deref) an atom, while the state/world keeps moving forward. I don't think any amount of tooling can solve that case for mutable data. A snapshot of a mutable data structure would require copying the whole data structure while using some sort of a locking strategy to stop writers while the read is taking place.
I also have a separate service that the server makes calls to, which doesn't run on this server in production (it has its own production server), but does run in dev and test. Each dev/test system runs a separate instance of this service, which has its own separate connection pool(s), and setting this up was trivial.
Needless to say, failures are reproducible and meaningful. There is no mocking -- we test against real local services with real local DBs. (There are still some remote service calls which I'm slowly replacing, and some flakey, unavoidable remote dependencies in a few browser tests).
I didn't do anything special to make this possible other than naming the config files "service-name-config" instead of just "config". It is just the natural result of passing state in explicit arguments. The same is not true of global state.
> It is just the natural result of passing state as explicit arguments.
But nothing you've mentioned here is intrinsic to mutable state. It seems like all that's happened is you identified a part of your program that you wanted to be configurable and exposed a configuration knob. If for example you wanted to make it so that there is a test mode that where you want to prefix "test-" to every string written to the DB that would also probably involve a new argument somewhere. There's nothing here special about the mutable state part of it.
I’m fairly sure this great quote is about mathematical “objects” in that you will never be able to truly “understand” or have a “real feeling” for more complex ones, like higher dimensions. Yet, by applying some simpler rules we can use and transform them, and after a bit of practice that will make it feel “close to us”, or “real”.
> Simplicity (of finite systems) is ultimately a function of familiarity.
I really don’t believe it would be true. Maybe I’m misunderstanding, but no matter how familiar I am with a given crud program vs JIT compiler technology, the latter will always be complex - but as you later refer to, I’m sure you know the difference between essential and accidental complexity. But in this view I would rather say that simple things are ones with minimal accidental complexity, while the easy-hard axis is about the essential part of that, that is irreducible.
>>> It all depends on the person.
Based on what I've recently learned about neuroscience and optogenetics, I don't think there's much evidence to support this sort of relativism. On the contrary, many processes in mammalian brains have common mechanisms.
To explore more, this is a great podcast https://peterattiamd.com/karldeisseroth/
Disclaimer: I am a complete layman on the topic, so please correct me if I'm wrong.
This bit is wher ehe says that about magnets: https://youtu.be/P1ww1IXRfTA?t=1300
While the 'simplest' (in the physics sense) description of something is elegant, it can also be extremely hard to understand and work with. Maxwell's equations are used in engineering for a reason - and not their simpler theoretical physics underpinnings.
Hickey's fundamental contention is that whether something is easy is an extrinsic property whereas whether something is simple is an intrinsic property. Whether something is easy is dictated often by whether it is familiar, whereas simplicity lends us the more ultimately useful property of being understandable.
To which I'll counter with Von Neumann's famous quote about mathematics : "You don't understand things [simple]. You just get used to them [easy]."
There is no fundamental difference between ease and simplicity. Simplicity (of finite systems) is ultimately a function of familiarity. There's a formal version of this argument (which is effectively that most properties of Kolgomorov complexity when applied to finite strings are defined by your choice of complexity function, even in the presence of an asymptotically optimal universal language. In particular there is not a unique asymptotically optimal universal language, that is the Invariance Theorem is overhyped), but the informal version is that both simplicity and easiness arise from familiarity.
Indeed the fact that there is "ramp-up" speed for simplicity suggests that in fact what is going on is familiarity. E.g. splitting state into "value" and "time" is one way of thinking about it. But I could easily claim that in fact "time" complects "cause" and "state." Rather state machines where the essential primitives are "cause" and "effect" are the proper foundations from which "value" and "time" then flow (you can think of "effect" nondeterministically, a la infinite universes, and then "value" and "time" fall out as a way of identifying a single path among a set of infinite universes). Likewise Hickey claims that syntax mixes together "meaning" and "order" whereas I would could just as easily say that "order" complects syntax and semantics!
What of the idea of "being bogged down?" That "simple" systems allow you to continue composing and building whereas merely "easy" systems collapse and are impossible to make progress on past a certain threshold? I claim that these are not intrinsic properties of a system. They are rather extrinsic properties that demonstrate that the system no longer aligns well with the mental organization of a human programmer. However this is dependent on the human! A different human might have no problem scaling it.
Now hold on, perhaps, while simplicity is perhaps dependent on the human mind and humans all more or less have the same mental faculties. Perhaps we can't find a truly intrinsic property that we call simplicity, but perhaps there's one that's "intrinsic enough" and relies only on the mental faculties common to all humans. That is, returning to the idea of "being bogged down," there are systems whose complexity puts them beyond the reach of all, or at least most, humans. We can then use that as our differentiator between "simple" and "easy."
To which I would reply that this is probably true in broad strokes. There are probably systems which are are so arcane as to be un-understandable by any human even after a lifetime of study. But at a more specific level, the way humans think is very varied. The ways we learn, the ways we develop are hugely different from person to person. Hence I find this criteria of "bogging down" far too weak to support Hickey's more concrete theses, e.g. that queues are simpler than loops or folds.
When you're talking about things like love, hate, and fear, sure maybe those are universal enough among humans to be called "objective" or to have associated "intrinsic properties," but when you're talking about whether a programming language should have a built-in switch statement, I don't buy it.
For the purposes of programming languages, simple is not made easy. Simple is easy. Easy is simple. The search for the Platonic ideal of software, one that relies on a notion of intrinsic simplicity, is a false god. Code is an artifact made for consumption by humans and execution by machines and therefore any measure of its quality must be extrinsic to the humans that consume it.
Sometimes X is simple. Sometimes it's not. It all depends on the person.
As empirical evidence of this I leave this final exchange between Alan Kay and Rich Hickey where the two keep talking past each other, no matter how simple their own system is: https://news.ycombinator.com/item?id=11945722