Distributed Erlang/Elixir has known limitations. For example, the network is fully meshed, which gives you about 60 to 200 nodes in a cluster. Or don't send large data over distribution, as that delays the other messages, etc. Some of those are easily solvable. For example, you can rely on your orchestration tools to break your clusters in groups. Or you can setup an out of band tcp/udp socket for large data. Others may be more complex.
The important question, however, is how far you can go without having to tweak, and, once you reach those roadblocks, how well you can address them. In many platforms, writing a distributed system is a no-no or, at best, they require you to assemble and tweak from day one. In this case, the ability to start with Erlang/Elixir and tweak as you grow is a feature.
And if you never run into those roadblocks, then you can happily continue running on the default stack. Just look at the many companies using Phoenix PubSub and Phoenix Presence, both distributed, without having to worry about fine-tuning the distribution.
https://www.manning.com/books/the-little-elixir-and-otp-guid...
A better architecture for a distributed system has a strong composability property. It should be possible to modify every possible aspect of it on a running cluster without introducing downtime.
Write your own standard well documented distribution layer and become independent of underlying technologies.
You can move away from fully meshed for topologies but how does this choice affect node ups and node downs? Rebalancing can affect how you store data in the cluster.
How many connections should you have between nodes? A single connection makes the ordering guarantee straight-forward. Multiple connections is more performance but requires care if you need ordering and is more efficiently done along side your application.
And what about process placement? On which side of CAP do you want your registries to sit?
If you and your team is capable of "writing your own standard well documented distributed layer" upfront, then you are in a better position than most to take those decisions. But writing a distributed system is hard, so I will gladly start with a well-designed system, especially at the beginning of the project, when it may be unclear which patterns I will need as my application and business grow.
And most times, it will be good enough.
As far as OTP goes, you can plug your own discovery/topology mechanism as well as your own module for handling the connection between nodes. But, as mentioned in my previous reply, some of those issues may be better solved on the side, e.g. a different tcp/udp connection for data transmitting.
But indeed, you need to know your tools like every other part of computer science. Storing 9 billion elements in an array, doing linear search and then complaining your linear search is too slow and needs to run faster will be disastrous to your architecture. Likewise, assuming communication is free is equally disastrous.
The problem with building a distribution layer in another language is the effort involved. Most of the companies which pull that off and get stability in addition are usually large, multinational, and has ample amounts of engineering resources to throw at the effort required.
That said, good Erlang architectures definitely knows when you should fan out and use a common layer for distribution. At $work we have a Kafka cluster for some event handling simply because it is the right tool for the job, and because it makes it easier to interface other programs on top of it.
In my experience, the distribution features are best used as orchestration however. Your distribution layer is Erlang, and then you use other languages for "leaves" in the architecture. If the right library is present in e.g., Go, then by all means use it. Erlang easily speaks protobuf for instance.
You can move it out just the same as you can elsewhere, you just add an unnecessary layer of complexity and lose a lot of the capabilities already built into the language. It boils down to a simple question of, what would be the purpose behind that decision? What improvement would you get from deciding to move any of that elsewhere? Do you gain speed at the cost of complexity?
Clustering is a non-trivial problem. It can't happen naturally in any language that includes mutable data without explicit oversight and thus most of the needed tooling can't be built into the system and guaranteed to work everywhere. As soon as some part of the system goes from "send message, get response" to "send message, change variable referenced in memory on this machine, get response" you break your ability to naturally cluster.
That leads to a dependency in central relay points for things like setting up web sockets and then having code behind them communicate. The code behind them in whatever language is usually going to be talking to other central relay points like load balancers or pub sub databases rather than directly to other specific servers.
As soon as you have central relay points, they become something else that you have to monitor and scale even if they are very high volume.
There's nothing wrong with this approach and it will certainly work but if you want to avoid that and build a distributed system in another language, you've got to invest a lot in other areas.
It is supposed to be easy. But they are not using language agnostic messaging. Instead they rely on Erlang distribution protocol. This is what makes it difficult to integrate.
> what would be the purpose behind that decision?
1. To achieve composability. I can replace one web socket server with a C++ implementation to see if I can handle 10 million connections per server. If I fucked it up return back to Erlang implementation otherwise fuck BEAM. I'm moving to native code. The beauty of it is that I can do it one step at a time. Even I can implement web socket layer in multiple languages and experiment all possible options in production and no one even notices it.
2. I don't have to pay for 20 Erlang guys who are mostly senior devs. Get 5 of them to write the core and the rest of the team NodeJS guys.
If you're going for your C++ layer, try https://github.com/saleyn/eixx for example.
You prefer doing all the work mostly from scratch, upfront and straight away, instead of having to maybe adapt a library later if you actually need to migrate.
I think your cost analysis is off.
1) you are misunderstanding the grand parents points. Better to have primitives that scale you to X rather than having to get to X on your own.
2) The grand parent wrote the Elixir language.
3) Making sweeping statements like "A better architecture for a distributed system has a strong composability property" is very easy.
For a fast time to market, I do agree with you. But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you.
> 2) The grand parent wrote the Elixir language.
I didn't know him, nor I care.
> 3) Making sweeping like "A better architecture for a distributed system has a strong composability property" is very easy.
I'm considering this as an excuse, than a technical argument.
This requires having distributed system experts on your team or having the money to call them from day one, before you even reach the market and before you are even sure you will have custom needs. If you have the need, the expertize and the time, then surely. But writing a distributed system from scratch should certainly not be the first choice (IMO).
Sure, you want a beautiful lover, but is your object of desire within your reach? One needs to be practical when playing engineer.
Let's try this: Composibility is a highly desired property for distributed systems however correctly scaling composible semantics is a hard problem.
This sort of duality props up all over the place in general engineering, and certainly in software systems engineering.
Fast forward several years, and isolated processes turns out to be a great fit for large SMP systems, and Erlang/beam is now doing signal path work in a lot of places. Erlang tends not to put explicit limits, but some techniques are going to fail at large scale; ex: if you have 50,000 processes across many nodes, sending the same message to each of those processes is going to be slow; sending one message to each node and fanning out from there is going to be faster; in no small part because you've reduced the network bandwidth you're using.
The nice thing when you hit Erlang scaling limits is that almost everything you need to fix is going to be in a pretty simple state. You're not going to find many things that are layers of optimizations on top of hacks on top of optimizations --- they do a good job of keeping things simple, and not optimizing until it's needed (and even then, they usually pick simple optimizations). Keeping things simple goes a really long way (especially with today's enormous servers).
Edited to add: I don't think they've even needed to tweak the vm yet either, just their user space code. That's pretty huge too.
Sending messages to processes on other nodes has the same syntax as sending to a process on your node, which makes it easy to run a distributed system. (Ports are different, you'd have to setup a proxy process on the remote node in order to send/receive from that).
Of course, with the base of process to process messaging you can build a higher level messaging queue (see RabbitMQ for a popular message queue built in erlang).
We haven't really had to. Really only args we use are "+sbt db +zdbbl 32000 +K true" and increasing the default process limit.
The true failure of the platform happens when you cannot do these tweaks and adaptation, or that their cost is shadowed by having written it in a more appropriate technology at a lower time/cost/effort budget.
In truly amazing open source fashion, they also made some libraries for other companies to leverage!!! Super big props to Discord for that one. Seriously can't thank their team enough for going above and beyond.
Under what circumstances does it get triggered (since you seem to be more knowledgable about this than I am!)? I expect records would not fall under this.
But it doesn't help with cases where you are constructing a term (dynamically) in a process and sending that term. There is more meat in the blog post of mine: https://medium.com/@jlouis666/an-erlang-otp-20-0-optimizatio...
Erlang was never about speed, it's about reliability, availability, and ease of concurrency. And Elixir just makes it easier to access those tools.
I guess the message here is, "There's never a magic bullet, but you've got everything you need to make yours."
Give that the whole selling point of Erlang/Elixir is scalability at the price of the rest, the article is really telling me to avoid the tech.
They likely have much more than 5 million users.
To put this amount in perspective, you get 5 million connections after receiving 3000 connections per second, which are never dropped and remain connected, for ~28 minutes. The majority of websites do not get even 300 requests per second.
Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language.
About the only thing I found irritating was a lack of language support for mutable hash tables, but I thought they fixed that at some point.
And Erlang's bit packing/unpacking borders on magical in terms of expressiveness and speed. I wish other languages would adopt it.
I think it's a lot harder to sell people on a language that looks syntactically unintelligible to a lot of web developers. Most people I know working with Elixir now are/were Rubyists, so it's easy for them to make the leap from that and start to understand all of the awesome things they get from BEAM and OTP.
Then again, FP for me was not difficult at all, and actually made several of the issues I had with my Ruby code not being expressive/obvious enough, or difficulty in composition, disappear overnight.
I've noticed that pattern-matching and guards just completely eliminate a stupid* amount of boilerplate logic in Ruby code
* "stupid" in hindsight, of course
With that said, I could write basically anything Elixir gives me with Ruby. The real value I get from elixir is the blazingly fast performance of BEAM and the speed at which I can build complex and stable distributed systems with OTP.
... I'm not like that. I took one look at Elixir and said "HOLY CRAP IT'S A FUNCTIONAL SCALEABLE RUBY!" (of course that was merely scratching the surface, but...)
Robert Virding, a co-creator of Erlang, creator of LFE, also wrote Luerl - Lua implemented in Erlang.
So if syntax is important you have many choices to use the BEAM VM/OTP in a Ruby-like language (Elixr), a Lisp (LFE) or in Lua (Luerl)!
I am now looking at Robert's videos of a game whose logic is written Lua, but it runs an Erlang process for each ship of thousands he spawns in the demo. Pretty cool. There's even Torchcraft - Luerl and Torch ML for learning AI/ML in Starcraft.
I still prefer LFE because I like Lisp syntax, biased as yourself about such things, however, I still think Elixir macros are not as far reaching as LFE's since they are handled after parsing in Elixir vs. LFE [1], although I wouldn't mind being proved incorrect on this.
[1] https://groups.google.com/forum/#!topic/lisp-flavoured-erlang/ensAkzcEDQQI think this kind of attitude is unhelpful at best, and alienating at worst. There are a ton of "serious" programmers, myself included, who care quite deeply about the enjoyableness of the tools we use, and aesthetic/expressive qualities absolutely come into that.
I believe the recent popularity of elixir kind of proves the case. There are many improvements to the package managers & tooling, etc, but the most obvious is to the syntax - which transforms what previously seemed unapproachable into a genuine option. Dismissing any and all such interest as merely "superficial" seems uncharitable, to say the least.
It is now getting better. But it was a really steep curve to adoption.
Call me unserious or shallow or whatever, but for me - it was a tiny bit about the syntax. More than a tiny bit, actually. The other things you mention, all valid, were just additional barriers to something already fairly aesthetically unpleasing.
It's improved and improving rapidly, as you say, which is really great to see.
It's not. Not for me at least. I prefer Erlang. Maybe I am strange like that.
I found initially the core concepts are hard - that is using using processes for concurrency, functional (immutable) data structures, functional patterns like recursion instead of for loops, the library ecosystems those are harder things. Those are the same in Elixir as well.
Erlang the language itself also simple. Think a bit like C and C++ if you're familiar. Erlang is like C, the language spec is small. Elixir has additional features which make it more expressive but also more complicated (macros, pipes). It's a bit like C++ having templates and classes. You can do more and with them, but it's a also a bit more to learn. I am exaggerating as Elixir is a lot more elegant and consistent than C++, I am just using the analogy to illustrate the idea of simplicity vs power.
Few projects actually need the level of scalability that Erlang brings, and the cost of building and running an Erlang project as an enterprise application is non-trivial. People need to understand this if they want Elixir to succeed where Erlang hasn't.
In turn, you can focus your development effort on new systems, new features, and scaling existing infrastructure since you are not bogged down in maintenance mode. Even better, you can often postpone errors in a system which are non-fatal. I often wait until I have a couple of bugs in a system and then I'll work on that system for a few days fixing them all. In systems where even the smallest amount of brittleness destroys them, this isn't really an option.
Another experience of mine is that Erlang systems tend to take a bit longer to write compared to cranking out a fast solution in another language. But the payoff is that your Erlang solution tend to have better robustness as load increases and people start using the system.
The community is small however, so you'll need to write more code yourself in-house as a result. For some problem spaces, this will hurt a lot because there are off-the-shelf solutions in other languages. On the other hand, larger projects which are more specialized can benefit from you knowing your own code better.
Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).
That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, and there is often one that gets me most of the way to where I need to go. Wrapper libs are nice sugar, but they're almost always unnecessary when you can just call Erlang directly (String v charlist conversions notwithstanding!).
But then the question is... how do you find high quality libraries without something like download count? The issue is signaling good libraries, with poor discovery. I subscribe to many "libraries worth looking at" mailing lists, the awesome lists, etc. but that is still a pretty coarse net. Searching for libraries on GitHub if you don't know the name already or don't really know what you're specifically looking for is an even worse experience.
Combine that with relatively poor fuzzy search on Hex, where even if you DO know what you're looking for, it leads to a very subpar experience. For example, "e-commerce" leads to different results than "commerce". [0][1] These then exclude [2], which is an ecommerce library available on Hex without that keyword.
[0] https://hex.pm/packages?_utf8=%E2%9C%93&search=ecommerce&sor...
But regardless of how good search is it wouldn't find ryal_core when searching for "commerce" if that word or any similar is nowhere in the metadata. The description for the package ryal_core is "The core of Ryal." which doesn't do the search any favours.
I think indexing GH READMEs would go a long way to helping the discovery problem, perhaps utilizing the new GH tags thing in some capacity, even letting authors tag their own packages. That would help in this specific case though this is of course a pathological example I happened to find recently. Maybe also having a "people who have X as a dependency also tend to download Y", though I'm not sure if you have that information outside of direct dependency graphs on Hex packages.
I feel like many newer languages have this problem, but the weird thing about Elixir is the 20-30 years of Erlang libs that are out there that are great that people have issues finding. It's worse if the libs aren't on Hex, and anecdotally I think people have been moving to Hex more lately for Erlang libs.
Building a solid standalone monitoring/metrics package is apparently next up on the "todo list" for Chris McCord (creator of Phoenix): https://www.youtube.com/watch?v=pfFpIjFOL-I
https://gist.github.com/asummers/3479eb6e3914d9b9a24c9566468...
However, the BEAM is not very fast on executing actual code, it is like Python in that way where it is good to slave out CPU-heavy work; the BEAM is built for async IO, and yes, running a JVM (or C++) system as a node on a BEAM mesh is fantastic for that (or a port, or NIF for small work).
If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-proven tools to save you massive headaches and costly rewrites.
Still, I feel, that using Elixir is, today, still a large bet. You need to convince your colleagues as much as your bosses / customers to take the risk. But you can rest assured it will not fail you as you need to push it to the next level.
Nothing comes for free, and at the right scale, even the Erlang VM is not a silver bullet and will require your engineering team to invest their talent, time and effort to fine tune it. Yet, once you dig deep enough into it, you'll find plenty of ways to solve your problem at a lower cost as compared to other solutions.
I see a bright future for Elixir, and a breath of fresh air for Erlang. It's such a great time to be alive!