It's amazing at what it does - If you had lots of physical nodes and needed to build a robust, fixed-size system at any moment in time, it's an incredible option.
But with container orchestrators, you get the ability to design a polyglot with more fluidity in your design. The admission that different languages are best suited to different tasks throws a monkey wrench into the Erlang ecosystem, because now I all of a sudden can't use Erlang for everything.
It does sound like an excellent choice for a control plane in many cases, however. I think it's phenomenal at what it does, but I think you have provided excellent context regarding its ideal use case.
The other commenter said it best: "But at the end of the day, Erlang is one technology that is good enough for most and does not require large operational experience to run. Your entire stack is in your own code, understandable by your entire team."
So the difference is doing your scaling and distribution natively in code, versus hiring and maintaining devops expertise to manage your containers. On top of that you now have one more "node" in your team (whether a single devops engineer or a group of them) that needs to collaborate with everyone.
I would note that Erlang is pretty good at absorbing other things into the Erlang abstraction. Depending on your needs, you can use:
• NIFs (library code loaded into the Erlang VM — good for performance, but not good for fault-tolerance)
• Port programs: arbitrary POSIX processes (as long as they speak a specified protocol on their stdio) spawned by the Erlang VM and communicated with + managed through a fancy socket abstraction (Erlang ports).
• Erlang "C nodes": arbitrary software that links in Erlang's erl_interface distribution-protocol library (https://erlang.org/doc/apps/erl_interface/ei_users_guide.htm...) as C code, and so can then join an Erlang distribution-set as a full peer.
That last one is especially interesting if you're building a polyglot system where Erlang is just one component: it allows you to build an Erlang "node" entirely in C or Go or Rust or Java or Haskell or whatever, that — from Erlang's perspective — can occupy one of the node-roles in your Erlang system architecture; while from the other software's side, can treat the Erlang abstract-machine as a guest within some other abstracted system. Through the erl_interface library, the C node gets access to all the interesting distribution and fault-tolerance tools of Erlang (e.g. it can interact with other nodes' ports; it can hold monitor refs; etc.) So you still can "use Erlang for everything" (related to distribution and fault-tolerance, at least) even in your other software.
This a great way to make things:
- You have N Cores on machine
- You create N Threads to match, you don't "create/destroy/create/destroy", no, you already have the size of the problem set!
- You assign roles, maybe just split between "heavy" and "light"
- Your context switching is for computing small things on light
- You do sequential code, tight loops and heavy I/O on heavy
I truly wish I could do something like this on most languages (ie: is part of the paradigm/semantics). I use Rust now, and async is great but the mental model is HARD.
- You don't have a easy way to split between workloads, anything could suddenly stall
- When is easy to launch async/goroutines you launch them like is no tomorrow.
- So, this mean I don't know what are the size of my threads or task are(could) be running without manually inspecting or assume.
- I don't know, after look at async function, to know which workload is for (without inspect the code and the flow of it) (ok, in rust you can maybe infer if you are clever in the use of types to be markers... but still is disconnected to the run of the program)
"container orchestrators" is like put a hammer to handle another hammer ("heavy OS + VM/DbEngines") under the pretense of being "light" and efficient, when actually this is how most software stack are:
- A "heavy" DB engine - A "light" logic server that connect to it - MAYBE out-of-band processing for send emails and stuff
So, the thing is, having Kubernetes or anything like put the solution too far away. I need, as developer, to MODEL that relationships and KNOW in what workloads this or that must operate. And the most fixed/predictable is it, the better.
You are then adding some redis/memcached, kafka or something similar, etcd perhaps and kubernetes.
A bunch of technologies your developers need to use properly and be running to develop on.
As opposed to having an easy to use single technology named Erlang which is almost certainly good enough for your needs. And your developers can easily be running a distributed system of erlang nodes locally without any issues or complex setups.
And anecdotally, for me who never had the need to use distributed erlang in practice, the benefits lay completely elsewhere. Just looking at programs written out there in the wild, some ubiquitous, and knowing the mess that is software development makes one wonder.
People try to scale Erlang systems by trying to get one distribution set to have thousands of nodes; and then taking that thousands-of-nodes cluster, calling that the “master” cluster, and then having another thousands-of-nodes cluster and calling that the “hot standby” cluster. But that’s precisely backwards.
Think like Ericsson. You’re designing a switch. You have two independent problems:
1. How do you make your switch Highly Available / fault-tolerant?
2. How do you handle higher load than one switch could possibly handle (for either hardware or software reasons)?
To solve problem 1, you create nodes, with roles (e.g. a master and hot standby), where there are an exact number of pre-defined nodes, each with exact static roles. These nodes have defined fault-tolerance relationships. Databases like Mnesia that replicate from one node to another. Applications that have defined chain-of-command such that it’s clear which node becomes the leader if the first leader fails. Etc. You design your system so that some of these nodes can go down, or netsplit, without the system itself going down/crashing. And then you deploy these nodes to separate machines, so that hardware failures will end up being treated the same as node failures.
To solve problem 2, you stamp out copies of this entire system of nodes, and add more infrastructure external to it to route between them.
In this model, a system of nodes — a distribution-set — is the same type of thing as a Kubernetes Pod. It’s several static processes, one of each role, that inter-depend, and are “wired” to one-another; and then get managed together / treated as one workload. The big difference is that, in contrast to K8s pods that all must get scheduled to one host together, Erlang distribution-sets usually get “scheduled” across several machines. But they still have one identity. They’re a single abstract machine that happens to consist of workloads running on multiple physical machines; where those workloads going down is a designed-for and tolerated aspect of the abstract machine they compose.
Everything that happens inside that single abstract machine, is the purview of Erlang and Erlang’s stdlib. Application failover, Mnesia, the thin-client code_server: they’re all designed to make a single abstract machine out of a defined set of Erlang nodes where each node has a static role.
Everything that happens outside of the abstract machine, is the purview of user code. Want to horizontally scale the abstract machine? Don’t use the distribution protocol; it’s not tuned for that. Use a userspace library. Have an application that does abstract-machine-to-abstract-machine peering. Make it the role of one or more of your nodes to run that application. Make it fault-tolerant, too.
Under this model, I hope you can see that statements like this are mostly incoherent:
> Erlang's not the slowest language, but it's not very fast, and using something 5-10x faster can "scale" nicely too by needing that many fewer systems in the first place.
...because an Erlang system architecture that builds one Highly Available abstract machine out of five-to-ten nodes, cannot be swapped out for a system architecture composed of one node, no matter the language it’s written in.
The other system, to implement the same fault-tolerance strategies, will need exactly the same number of nodes, because each node performs one or more roles, and serves as a VM-level failure-kernel / bulkhead for HA within those roles.
The non-Erlang-implemented system will just have a harder time implementing such strategies, because it’ll mostly have to use userspace libraries/frameworks — rather than VM-level abstractions — to do so. (Though you can get close with languages that have basic syntax that compiles down to framework code in other languages, e.g. if there were a hypothetical “Erleans” that compiled down to C# + .NET Orleans framework calls.)
——————
Of course, none of this matters if you don’t have the problem that Erlang solves. If you don’t need a stateful abstract machine that keeps itself online and computing over that state at all costs with fancy multi-node footwork, then you really don’t need any of Erlang’s distribution stuff.
But there are still, today, problems that do put you in that position. Stateful packet switching is still a thing. Game servers are another good example. App-layer cache servers (Redis, memcached, etc.) where your architecture couldn’t survive the thundering-herd that would happen if you needed to re-warm the cache from a cold start. Etc.
In those contexts, do you really want to try to jury-rig a solution using e.g. Hazelcast + IPVS failover? Better to just use Erlang. Because it’s still ahead of its time, for this problem domain.