News for hackers. Stuff that matters.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Wednesday November 23, 2022 @06:58PM (#33726103)
The BEAM is awesome, but both Elixir and Erlang are not really great for concurrency. They lack support for pure functional programming, which is really one of the most useful things in this area.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by throwawaymaths on Wednesday November 23, 2022 @07:42PM (#33726465)
More important than pure functional programming is: smart timeouts, builtin load shedding/backpressure, supervisory restart trees.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Wednesday November 23, 2022 @08:35PM (#33726874)
That's like saying "more important than food is having a full stomach". Pure functional programming is a tool (or paradigm) that makes the things you mentioned extremely easy and safe to implement and extend. It focusses on the local runtime though, not on distributed systems (where the BEAM comes in) - which is why they don't contract themselves.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by throwawaymaths on Thursday November 24, 2022 @01:09PM (#33734420)
BEAM is not a pure functional language system. There's lots of hidden state: message passing, IO, process dictionary, ets tables, counters, sockets, NIFs, etc.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by brightball on Wednesday November 23, 2022 @07:04PM (#33726164)
How are Elixir / Erlang anything but great for concurrency?
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Wednesday November 23, 2022 @08:32PM (#33726856)
Using Elixir / Erlang on the BEAM is great for concurrency - but the programming languages themselves are not. It's an important distinction, because I believe that there is still a lot of potential for those languages.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by ramchip on Wednesday November 23, 2022 @08:44PM (#33726954)
What is it about pure functional that makes concurrency much easier? Is there a code sample that would illustrate it?

I've done both Haskell and Erlang/Elixir and yet I don't really see what you're referring to concretely. There was hope that automatic parallelization would be a big win for pure functional, but I don't think it's really worked out in practice, because of the overhead and difficulty of predicting if parallelizing something would make it faster or slower.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Wednesday November 23, 2022 @10:38PM (#33727567)
It's not about performance as much as correctness and easiness to understand code. Functional programming removes the need to simulate (local) state in your head when trying to understand what code does. And in the places where it is inevitable, it makes it explicit and leads to a design where it's easy to track why and how things got changed.

For example, .map and .filter are now in almost every language, because they are much easier than for-loops. You see them and you know "aha, this collection will be transformed, the number of elements stays the same" or "this collection will be filtered and the elements will look the same, but some might be gone and no new ones will have been added".

Pure functional programming is similar, just that the scope is suddenly "the whole runtime/machine" instead of "a small piece of code that somehow does a loop".

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by ramchip on Wednesday November 23, 2022 @11:42PM (#33727880)
If I understand correctly, you're saying Erlang/Elixir code would be easier to understand and work with if functions that did I/O or message passing etc. were labeled as such, via type info.

I don't entirely disagree, but I also don't think it would be a game changer for a couple reasons...

- It's already well established practice to separate pure and impure modules in Erlang/Elixir projects, even without a type system to enforce it. Take Ecto for instance, it has a clean split of side-effects (Ecto.Repo) and pure logic (Ecto.Changeset). Very different from things like ActiveRecord or Django ORM. [1]

- BEAM programs (and libraries) use a lot of concurrency and message passing, and I think you would have to use an escape hatch ala unsafePerformIO more often than a typical Haskell program, otherwise the IO would "infect" most of the code. Things like instrumentation, fetching app environment, calling the code server, using process dict...

Joe Armstrong kind of mentions the latter problem in his thesis [2]:

    Notice that I have chosen a particularly simple definition of “dirty.” At first sight it might appear that it would be better to recursively define a module as being dirty if any function in the module calls a “dangerous” BIF or a dirty function in another module. Unfortunately with such a definition virtually every module in the system would be classified as dirty.

    The reason for this is that if you compute the transitive closure of all functions calls exported from a particular module, the transitive closure will include virtually every module in the system. The reason why the transitive closure is so large is due to “leakage” which occurs from many of the modules in the Erlang libraries.

    We take the simplifying view that all modules are well-written and tested, and that if they do contain side-effects, that the module has been written in such a way so that the side effects do not leak out from the module to adversely affect code which calls the module.
He is of course talking about "dirty" _modules_. Working at the function level, the leakage wouldn't be quite as bad... but I think it may still be enough to limit how useful IO annotations would be. Code that "has been written in such a way so that the side effects do not leak out" is quite common on BEAM.

Who knows though. Maybe we'll see interesting things in Gleam in the future :)

[1] "To spawn, or not to spawn?" is a good article on the practice of separating I/O (GenServer) code and pure code: https://www.theerlangelist.com/article/spawn_or_not

[2] https://erlang.org/download/armstrong_thesis_2003.pdf

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Thursday November 24, 2022 @08:44AM (#33731196)
> If I understand correctly, you're saying Erlang/Elixir code would be easier to understand and work with if functions that did I/O or message passing etc. were labeled as such, via type info.

No, not quite. Pure functional programming is a specific style or paradigm. It's about writing referential transparent expressions only. Tagging something is "impure" is absolutely not the same, even though doing so and/or separating pure and impure functions is a good first step on the way to pure functional programming.

> Who knows though. Maybe we'll see interesting things in Gleam in the future :)

Would be nice and might make me switch ecosystems. As of now, there don't seem to be concrete plans:

> Yes, Gleam is an impure functional language like OCaml or Erlang. Impure actions like reading to files and printing to the console is possible without special handling. > We may later introduce an effects system for identifying and tracking any impure code in a Gleam application, though this is still an area of research.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by User23 on Wednesday November 23, 2022 @11:19PM (#33727755)
Speaking of important distinctions, concurrency isn't a property of a language, but rather a property of a runtime. It's the old procedure vs process distinction. Now a language can have nondeterministic constructs that beg for a concurrent runtime, but that doesn't actually require it. After all, a nondeterministic construct can be implemented deterministically. If the language doesn't specify the order the nondeterministic cases are handled in, then the runtime is free to handle them in some deterministic way.

Incidentally that's the root of the misconception that pure functional languages are easier to reason about. It's operational reasoning that's problematic, because simulating a nontrivial state machine in one's mind is a great cognitive challenge for those of us who are less bright than John von Neumann. However, the Dutch basically solved this problem and provided a logical framework for reasoning about imperative programs in a way based off the program text, with no need for attempting to mentally simulate the probably infinite set of possible processes it describes.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by nequo on Thursday November 24, 2022 @12:46AM (#33728130)
> However, the Dutch basically solved this problem and provided a logical framework for reasoning about imperative programs in a way based off the program text, with no need for attempting to mentally simulate the probably infinite set of possible processes it describes.

Do you mind clarifying what the framework is that you mean? A pointer would be helpful for me.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by brightball on Thursday November 24, 2022 @08:55AM (#33731301)
My guess is that he means OTP.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by User23 on Thursday November 24, 2022 @07:45PM (#33737803)
See Dijkstra and Scholten.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Thursday November 24, 2022 @08:46AM (#33731222)
> Speaking of important distinctions, concurrency isn't a property of a language, but rather a property of a runtime.

But the way of how you can and will describe how the runtime should execute code in a concurrent way is highly dependant on the language. Compare Assembly and Go or Rust or Erlang. Completely differnt worlds, don't you think so?

> Incidentally that's the root of the misconception that pure functional languages are easier to reason about.

It's not a misconception. People are different, but at least I can say after many years of experience in both worlds that pure functional programming is much much more productive than any kind of other approach that I have tried when it comes to dealing with concurrency, parallelism and effects.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by User23 on Thursday November 24, 2022 @07:49PM (#33737832)
> It's not a misconception.

I hear you, but you’re missing the point. The sole reason why you find pure functional programs easier to reason about is because that is one strategy for avoiding operational reasoning. However it’s equally possible to avoid operational reasoning about imperative programs. Hence my claim that it’s a misconception.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Sunday November 27, 2022 @09:31PM (#33769207)
I would be insterested to learn how this can be avoided without pure functional programming. Any sources/references?
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by latch on Wednesday November 23, 2022 @11:13PM (#33727718)
I've been doing Elixir professionally for 6 year. There's a whole class of problems that are best solved with concurrent read/write access to data. So the answer to your question is exactly that: when you want N-cores to work on the same piece of data.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by brightball on Sunday November 27, 2022 @03:09PM (#33766299)
I still don't understand how OTP doesn't already address that case?
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by slindz on Wednesday November 23, 2022 @07:19PM (#33726273)
Strong disagree.

Elixir has been my primary language since 2016 because it makes concurrency so easy.

I'm also a little confused how it's not functional enough for you.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Wednesday November 23, 2022 @08:36PM (#33726891)
Please mind the distinction between functional programming and pure functional programming - despite the name it's very different.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by dmitriid on Thursday November 24, 2022 @05:26AM (#33729687)
It's also the difference between being pragmatic and being in a cult.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Thursday November 24, 2022 @06:16AM (#33730035)
That sounds like you had some bad experiences - what happened?
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by dmitriid on Thursday November 24, 2022 @06:37AM (#33730189)
I've had a bad experience with zealots.

Your original statement lacks any nuance except "E are bad because no pure functional programming". Any attempts to clarify resolve to basic "it's for the good of the whole program".

Whereas easy-to-use concurrency primitives, effortless parallelisation and concurrency, and even the most basic stuff like ability to put a logging statement anywhere in the code without re-engineering half of your program trump whatever imagined advantages of pure functional programming you may come up with. Any day of the week, and twice on Fridays.

In theory, theory beats practice. But in practice...

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Thursday November 24, 2022 @09:26AM (#33731588)
I did not intend to write an essay of why PFP would be great to have in Elixir/Erlang. I'm just posting my opinion here and I'm happy to explain it as you can see from my answers.

> ability to put a logging statement anywhere in the code without re-engineering half of your program

Let me ask you a question: does it change the semantics of your (whole) program if that logline, that you are talking about, is not executed for some reason - or if it is executed more than once?

If your answer is "it doesn't really matter, might at most be a bit annoying but it's just a log, no stakeholder of the program can ever notice" then there is no problem with putting this logline into the program. No need to re-engineer anything.

If, however, this logline is critical and will be e.g. parsed and used by another system and actions might be taken due to it, then I would argue it is good if you are forced to consider the potential impacts to your program. If that means that you need to re-engineer half your program then there is a good reason for that, since the potential impact is huge. Such a thing has never happened to me in many years while working on different kind of applications. Sometimes a couple of 100 lines need to be rewritten - that is the max that I ever had to do. And indeed sometimes this rewrite in fact caused me to find and resolve problems that would have otherwise been introduce by accident.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by dmitriid on Thursday November 24, 2022 @10:36AM (#33732500)
This amounts to demagoguery.

Because yes, it's a fact that in a "pure functional program" you need to re-engineer half of the program if you need to put the log somewhere where it's "oh so pure", and where you didn't need logging before [1]. Or thread IO through the entire program to begin with.

> Sometimes a couple of 100 lines need to be rewritten - that is the max that I ever had to do.

Where in a pragmatical language you just add `Logger.log` or equivalent

[1] Spare me the pontification of "if you need logging, you're doing something wrong, this must be covered by tests or type systems". There are things you must log like metrics, audit logging, tracing values through the system, and it is a 100% certainty that you will add logs to places where no logging existed before.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Thursday November 24, 2022 @07:30PM (#33737729)
You have completed ignored my question and the explanation of why I asked. I have to assume you just want to rant a bit here. Sorry for your bad experiences, but they are hardly representative.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by dmitriid on Friday November 25, 2022 @01:44AM (#33739911)
> You have completed ignored my question and the explanation of why I asked

I did not. I wrote that it amounts to demagoguery.

The reason is simple: a pragmatic language lets you write a single `Logger.log` line without pseudo philosophical discussions on the semantics of a program and "sometimes re-writing 100 lines of code".

It's no wonder any discussion on "how to do logging in Haskell", for example, devolves into discussing the merits of various types of monads and "composable co-monadic contravariants" with 15 equally cumbersome ways of using them.

"All for want of a nail"

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Friday November 25, 2022 @01:59AM (#33739967)
Not that I'm a fan of Haskell, but I believe that if this log is critical to the behavior of the program AND it is inserted in a place where important effects have previously not been expected (such as pure mathematical calculations) then it is a good thing that the language makes you aware of that. Someone might for example be caching/memorizing those calculations and suddenly your log isn't always executed and that might be a bug.

I much rather prefer to evaluate the impact in advance rather than having to find and figure it in production.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by dmitriid on Friday November 25, 2022 @03:02AM (#33740268)
> I much rather prefer to evaluate the impact in advance rather than having to find and figure it in production.

See, this is exactly the zealotry and demagoguery I am talking about.

99% of use cases: we need to add a single line of logging here

Pure functional programming cultists: first we must consider the semantics of the program and the implication of logging on the grand theme of things. Consider the criticality of a log line. What is a log line? ... <two hours later> ... an lo, once you've done the refactoring to consider the co-variants ... <another two hours later>

Edit. I'll reiterate:

Easy-to-use concurrency primitives, effortless parallelisation and concurrency, and even the most basic stuff like ability to put a logging statement anywhere in the code without re-engineering half of your program trump whatever imagined advantages of pure functional programming you may come up with. Any day of the week, and twice on Fridays.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Friday November 25, 2022 @06:00AM (#33741230)
If you need hours to figure out if a log line is relevant or not, then you have much more important problems to take care of.

For example, if this is a log line for audit logging in an enterprise product then it's clearly relevant. If this is a trace log line in case you need to debug some minor issue, then that's a different story. Has nothing to do with zealotry, just common sense. If you don't understand that there is a difference between those two cases then the discussion ends for me here.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by dmitriid on Friday November 25, 2022 @07:30AM (#33741784)
> If you need hours to figure out if a log line is relevant or not, then you have much more important problems to take care of.

I don't.

> Has nothing to do with zealotry, just common sense.

Ah yes. "does it change the semantics of your (whole) program if that logline, that you are talking about, is not executed for some reason - or if it is executed more than once?" vs. "this logline is critical and will be e.g. parsed and used by another system and actions might be taken due to it, then I would argue it is good if you are forced to consider the potential impacts to your program. "

etc. etc.

And yet the fact is that if you need to log something your precious "mah purity" function is doing, you're stuck with "sometimes a couple of 100 lines need to be rewritten".

Where as non-cultists just do a `Log.info` etc.

If you can't understand that, well :shrug:

Edit. I just re-read that inane pseudo philosphical bullshit about "the semantics of the whole program a log line". No, I definitely don't need to consider the semantics of the whole program to add a bloody log line.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by tome on Friday November 25, 2022 @11:35AM (#33744040)
> it's a fact that in a "pure functional program" you need to re-engineer half of the program if you need to put the log somewhere where it's "oh so pure", and where you didn't need logging before

It's most certainly not a fact. It's trivial to insert Debug.trace, for example.

Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by ilyt on Wednesday November 23, 2022 @08:12PM (#33726692)
Forcing functional programming upon developers instead just giving them tools to do it well is a disadvantage, not a feature.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Wednesday November 23, 2022 @08:33PM (#33726862)
I think you misunderstood me. I said "They lack support for pure functional programming" which does not imply that they should force the developer into using it.
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by ilyt on Thursday November 24, 2022 @08:20AM (#33731002)
What it is missing in Elixir/Erlang ? I only have ops experience with those
Re:Ruby concurrency is hard: how I became a Ruby on Rails co... by valenterry on Thursday November 24, 2022 @09:09AM (#33731409)
Basics for pure functional programming support would be some kind of IO-effect-type, tail-call-optimization (Elixir supports that), syntactic sugar for IO-combinators (to build up the "concurrent execution-plan" without actually executing it).

Optimally also stdlib functions and an ecosystem that help. Those can be rebuilt, but most people consider them tightly related to the language, so without it, it couldn't probably be called great support.