In particular, it is preemptive. This... Makes a lot of stuff easier.
Instead of running a single VM with full knowledge of how to run lightweight processes, designed to fully take advantage of modern multi-core CPUs with multiple guarantees enforced by the runtime you have multiple single-threaded VMs awkwardly communicating with each other over a bolted-on API
It’s complicated.
Node itself has never been single-threaded. The execution model for JavaScript is single-threaded, so there’s no working around that, but libuv uses threads to build async IO on top of blocking operations.
Then there’s worker threads, which are pretty similar to web workers AIUI, that give you parallel execution for cpu-intensive work.
Obviously, though, none of these facilities compare with BEAM
Lastly, the concurrency are primitives to the entire runtime, not a set of external libraries maintained by whoever, which might be incompatible with other libraries you might want to use.
But about the specifics of implementing a web-crawler, a NodeJS way to implement it would be to parallelize using lambdas.
To spread work over multiple cores with the cluster (or even worker-threads) modules you have to do so explicitely and manually. It’s essentially the same model you get with pthreads, or java, or python.
BEAM is a completely different model, the “processes” are internal tasks, which the runtime will schedule appropriately over available cores (the scheduler has been multithreaded for 15 years or so), spawning processes is as common as spawning tasks in JS, except each of these is scheduled on an SMP runtime.
beam will literally send processes between machines erlang) more easily than node will balance load over cores.
On a number of points the presenter was proposing, like message passing, immutable structures, and process tree management, the presenter would say, "but Erlang's had this feature for many years..." and the room would laugh and turn around and acknowledge Joe. He was modest but the validation must have been nice.
Go routines were pretty directly inspired by Erlang processes, so in terms of primitives I'd say they are very similar, aside from go lacking the distributed features you already mentioned.
Where Erlang/Elixir add value beyond go routines is what OTP (kind of the standard library) provides on top. Pre-built abstractions like GenServer for long running processes, Genstage for producer/consumer pipelines and supervision trees for making sure your processes haven't crashed and restarting them and their dependents if they have.
The devil, however, is once you go beyond the trivial.
First, the units of work operate completely differently, BEAM follows the actor model rather than CSP, meaning every actor has an address / mailbox and the actors can send one another messages through this, any actor can send any other actor (they're aware of) messages, and actors can process their mailboxes however they want.
But BEAM is also completely strict and steadfast about its actor model: its actors are processes, each has its own stack but also its own heap, when one actor sends a message to an other, the message content gets copied (/ moved) from one stack or heap to an other, processes do not share memory[0]. Incidentally this is what makes distribution relatively transparent (not entirely so, but impressively still): if everything you do out to interact with others is send asynchronous one-way messages, it doesn't really matter whether they're in the same process (OS), in a different process (OS), or on a different machine entirely.
The reason BEAM works like that however is not for any sort of theoretical purity, instead it is in service to reliability, which is the second big difference between BEAM and Go: BEAM error handling is inter-process, not intra-process. BEAM's error handling philosophy is that processes encounter errors, for all sort of reasons, and when that happens you can't know the entire state of the process, so you just kill it[1] and it should have one of its buddies which is linked to it, and whose job is to handle this and orchestrate what happens.
BEAM has built-in support for linking and monitoring. In the context of erlang, linking means that if one process dies (crashes), the other is sent a special message which also kills it. This message can be received as a normal message instead, in order to handle the crash of your sibling (in which case you receive various metadata on the crash). Monitoring means you just want to receive the crash signal. The reason you might prefer linking to monitoring is that if you're a manager of other processes and you crash, you probably want all the processes you manage to die as well. Which doesn't happen with monitors.
That is because BEAM has its origins in telecommunications, where reliability means redundancy, and oversight. So the way you structure an application in beam (often) is a tree of processes, where many of the processes have oversight of a subtree, handle fuckups (maybe by restarting, maybe by something else), serve as entry point to their workers, etc..., and if one of the leaves dies that's just a signal sent to its parent, which might just die and signal its parent, which will handle it somehow. This is the design principle known as the supervision tree: https://www.erlang.org/doc/design_principles/des_princ#super...
The third big difference is more philosophical and has to do with code reuse: because of (2) above, a lot of erlang / beam / otp is communicating between processes in a subtree, moving messages between them, exit signal strategies, etc... which leads to behaviours (https://www.erlang.org/doc/design_principles/des_princ#behav...), which are pretty alien because they're more or less mini frameworks, which not only are two things which are usually put opposite one another, but many people don't really want to hear about frameworks.
But that's what they are: behaviours are the encoding of entire prototypal lifecycle and communication patterns, where the user / implementer of the behaviour fills in the "business" bits.
Oh yeah and beam comes with an entire suite of introspectability tooling, which is kinda linked to (2): all the oversight thing ends up at people, so you can connect to a runtime and look at and touch all the things, more or less.
BEAM is a bit of an OS running on an OS really, probably closer in philosophy to the image-based languages of the 80s. In part because it is a language from the 80s. Not quite image based though, or in an other way designed to go even further and just run forever, as it includes built-in support for hot code reloading and migrations (though from what I remember that's not super great or fun, it was quite messy and involved to actually do properly).
By comparison to all that, goroutines are just threads which happen to be cheap so you can have lots.
[0] kinda, some objects live on shared heaps as an optimisation but they're immutable and reference counted so it's an implementation detail.
[1] and here if actors share any memory, an actor might be dying in the middle of updating or holding onto shared state, which means its error corrupts other actors touching the same state
It would be a dark day to discover my AWS t1.337metal was blocking 63 cores on I/O when nearly every modern lang has a wealth of async functionalities just awaiting to be exploited.
To junior developers reading this, such statements are never true.
According to others in this discussion they also made architecture changes (DB, Kafka etc.). Do we know if that improved the performance?
There is no objective way we can tell if Elixir had any performance impact. It could have been due to the rewrite, the architecture change or a combination of both.
Yeah the BEAM with some Rust NIFs is a great combo. I'd definitely consider it in the future for many types of problems and anything involving a HTTP or GraphQL interface.
Take the hot code reloading and actor model-based concurrency as a prime example. It's like getting AWS-level functionality without the steep bill for a lot of companies.
Though, I gotta admit, it used to be a hard sell for CPU-heavy workloads, especially number crunching. But Elixir is stepping it up with their Nx library, so that's changing.
Examples of companies cashing in on BEAM's efficiency:
Bleacher Report: Went from 150 servers down to 5. No joke.
Discord: Handles millions of real-time users without breaking a sweat or the bank.
Financial Times: Their content recommendation engine got both efficient and cost-effective.
Change.org: More petitions, fewer servers.
Podium: A million SMS messages a day and didn't have to massively scale hardware.what are "extremely powered machines"?
> It's like getting AWS-level functionality without the steep bill
which part of AWS functionality? load-balancing Beanstalk-style is free. AWS compute is not free, but neither is compute free with Elixir or whatever stack you run.
Similarly, distributed Erlang allows Elixir to run across multiple nodes. This could cut down the need for extra AWS instances or orchestration layers like Elastic Beanstalk. And when it comes to deployments, Elixir's hot code swapping can simplify what might otherwise require rolling updates or blue-green deployments with Elastic Load Balancers in the AWS ecosystem.
On the concurrency front, Elixir is designed for handling a high number of users and tasks simultaneously, which might reduce your reliance on EC2 or Lambda. Phoenix, Elixir's web framework, even has real-time capabilities baked in, so you don't need extra services like AWS WebSockets for that.
Finally, Elixir's actor model can serve as an in-memory message queue, which could potentially negate the need for something like AWS's SQS. So, while you're still incurring compute costs, the need for additional AWS services could be lessened, thereby simplifying your architecture and perhaps lowering overall costs.
Erlang and BEAM was designed to handle Ericson's telephony services. Piping-lots-of-stuff-in-parallel in fault tolerant fashion is the main use case around which it was designed.
Of course this is anecdotal.
When a person makes a claim, especially such a ridiculous one, it is perfectly valid to outright reject that claim without any argument. Why? Because no argument was provided in the first place.
Also I dislike FUD spreading so there's that, I am trying to engage them in a more-informed manner.
Though I am not sure whom you are addressing, my parent poster or theirs.
Hype and FUD are different sides of the same coin
Do you feel that Elixir is being needlessly hyped?
You'll note I'm not selling anything here, and no one is paying me a commission.
A junior that got "swindled" by my claims and spends a weekend learning Elixir becomes a better programmer and earns another feather on their cap. How tragic.
Junior devs, if you want to become a senior greybeard like me, learn anything that tickles your fancy, and ignore anyone that says it isn't worth your time. Even learning COBOL will make you a better programmer. I can only promise that Elixir is more fun than COBOL.
I write concurrent code without much extra effort compared to non-concurrent code. Elixir makes it easy.
Elixir/BEAM do have some benefits that are worth considering for many projects. But they absolutely are not special in this regard, and that's the junior-developer trap about which the person to whom you replied was referring.
The BEAM VM itself is a different beast compared to the JVM. It's more like its own mini-OS designed for real-time multitasking. Each process has its own garbage collection, and it's all non-blocking. So if one process goes belly-up, it doesn't take the whole system with it. Imagine a network of telephone switches; if one gets zapped by lightning, the rest keep chugging along. That's the level of fault tolerance Erlang and BEAM were designed for.
Now, speaking of fault tolerance, let's talk about how easy it is to mess up an Akka system if you're not careful. Say a new Java dev joins your team and doesn't get Akka's actor model. They might introduce shared mutable state between actors, which is a big no-no and can lead to all sorts of race conditions. Or they might do something like put a blocking operation inside an actor, which can hog resources and mess up the whole system's performance. Akka's great, but if you don't follow its principles, you can still shoot yourself in the foot.
So, the beauty of Elixir and BEAM is that a lot of these good practices are enforced by the VM itself. You get that fault tolerance and concurrency baked right in, without having to rely on every engineer knowing all the best practices.
Hope that clears things up!
In Elixir, on the other hand, you could create a module which is like a server process. You can start this server in a procedural flow, or you can "connect" it to a supervisor by giving it the startup information needed and a strategy to be used on how to restart the process in case it crashes for some reason.
A client process (or processes) with it's own module can then send messages to the server which will handle the incoming messages in its inbox sequentially. If you squint at it from an angle, modules look like classes in that they provide a way to separate code logically.
This way of doing concurrency takes getting used to and has a higher initial learning investment. But it feels cleaner and is less prone to user errors. In go for example you have to be careful of closures and shadowing which will result in shared memory and hard to debug errors, even though the initial investment in learning Go is much lesser.
Is it possible to get to the same result? Yes. It is not, however, anywhere in Elixir's ballpark of "easily". Do not discount the power of language-level, not just support, but encouragement. Especially when you're working with junior devs, if "the right thing" and "the thing the language wants you to write" are not in alignment, everything is much, much harder. Erlang and Elixir actively encourage easily parallelized code. Java activity encourages tangles of objects.
Erlang and BEAM, the things underlying Elixir, were specifically engineered to be used in a reliable, distributed fashion including a gigantic amount of idioms and support that no other language comes close to.
Erlang's bit syntax is hands down the best byte stream serialization that still exists--both from a perspective of performance as well as expressiveness. OTP is a documented set of idioms and behaviors for building systems that are meant to be highly distributed and deal with failure gracefully. These include things behaviors like in-place upgrades, supervisors which shut down and restart failing processes, error delegation, etc.
Erlang was meant for genuine "five nines" reliability. No other language comes close (maybe Ada does--I'll let their proponents chime in about that).
What you need to be fast is a systems engineer.
Not for long. Something is looming on the horizon in Java-land.
Only if you think that the BEAM is similar to being able to easily spawn a function on a separate thread and having channels.
Last I checked, goroutines had no mailboxes, supervisors, process monitoring, registry, and its scheduler has a much smaller scope and featureset than the BEAM's.
I swear it's obvious when people comment about the Erlang ecosystem without having really used it for anything.
BEAM's got this whole ecosystem built around it, right? Mailboxes, supervisors, process monitoring, and a registry—these are all first-class citizens in the Erlang world. And let's not forget the scheduler; it's like comparing a Swiss Army knife to a simple pocket knife when you look at BEAM's scheduler next to Go's.
It's kinda funny when people talk about BEAM and Erlang as if they're just another runtime or language. (it's more OS like than traditional VM like) They're really more like a whole philosophy of how to build robust, fault-tolerant systems. And if you haven't actually built something substantial with it, you're likely to miss out on what makes it so special.
Couldn't agree more with your take.
In my experience I think my statement still stands.
The key VM feature you're missing is process isolation. Without it, supervisors are not really possible - you can implement something that vaguely looks like them, but it won't provide the fault tolerance guarantees.
Imagine for example an Erlang process that leaks files. At some point it hits an EMFILE, gets killed, and the supervisor restarts it. The system will then go back to operating normally.
Now imagine a Goroutine doing the same. It hits EMFILE, exits, and the supervisor restarts it. This doesn't help anything: it just hits the same error and the system is unusable. There's no way for the VM to guarantee cleanup when a Goroutine exits, because it doesn't isolate Goroutines and track which one owns which resources.
Links and monitors are tools to extend the same behavior to user-managed resources like DB connections and so on. The responsibility for cleanup if a process crashes while holding a DB connection falls on the DB connection library, not on the error handling inside the crashing process.
Are you aware Erlang was designed to be a VM(the BEAM), a language for that VM(Erlang) and a comprehensive set of patterns and infrastructure for fault tolerance(OTP)?
All these as a whole are Erlang and the whole reason it has such reputation. I recommend reading https://erlang.org/download/armstrong_thesis_2003.pdf
> But they aren't core to how the BEAM handles concurrent processing.
They are, unless you want to reduce it to "it's just a pre-emptive scheduler with greentrheads".
> In my experience I think my statement still stands.
They're similar in that both have (mostly)pre-emptive schedulers, that's about it.
IO lists, which are the foundation of anything that build a string incrementally, are managed with vectored IO syscalls out of the box (readv/writev), which you'd have to handle yourself in most other languages, or resort to allocating and endless memory copying which Erlang is able to avoid.
I mean, if your business logic is inherently serial it makes little difference if you run it in a single thread or if each serial segment in between IO requests is run in a different thread. One way or another it's not going to get parallelized.
When would this EVER be the case? I'm struggling to imagine a scenario where logic doesn't/can't run in the worker thread.
There is a lot of focus on raw performance on web-related services, when in reality most of their running time is spent waiting for IO. If there are two things the BEAM excels at, is IO and turning almost any problem into half a dozen processes that are scheduled and run in parallel, if not geographically-distributed, with 1/50th the effort of any other language.
We live in a world with 32+ core CPUs. If your load is not spread uniformly all over those cores, you're losing a ton of performance. Handling requests over separate threads, like 99% of languages do, still isn't enough if all the business logic runs on the same thread.
I'm currently writing a web crawler in Elixir, and it is easier to design it so every request is done and processed in parallel, than to write a naive sequential one you'd do in any other language in half a day.