if you are going to rely on a stateful service to do this for you eventually the latencies will add up. for in-memory data, you're going to take a trip to redis and back. For persisted data, you're going to take a trip to postgres and back. Plus internal locks in those systems. Plus if you get bigger, you will need sharding logic, and maybe even another distributed shim (like galera or greenplum) on top, which adds even more latency.
For a real-world example of where supervision trees were very important, I once managed a virtual machine orchestrator I'd written in elixir, and they periodically polled the virtual machines to retrieve state. It's a nontrivial question, what do you do when the state of the system is not what you expect? (perhaps something IRL has changed though a side door) And what do you do if you have multiple concurrent independent systems that are dependent on that real-world state?
With a supervision tree, the solution is simple (1 LOC), you slay the parent supervisor and trigger ALL the relevant processes (and no processes, say, checking in on a different machine) to re-establish the ground truth. And you don't have to wait for containers to spin up, etc, because erlang processes are snappy, so there is less downtime. In the system I was managing there were many, many internal errors (netsplits, etc) and the customers generally didn't notice.
Also... the rhetoric here on HN about Erlang/Elixir vs. Kube really needs to stop. People are misinformed. They are not mutually exclusive, you do not need to choose one or the other. We need to reconcile that because the discourse always goes to this weird war in the comments.
[0] https://blog.plataformatec.com.br/2019/10/kubernetes-and-the...
If it's not a random example and if you're at liberty to say, would you mind elaborating further how Elixir specifically excels in that particular use case, and what would an alternative approach, regardless how cumbersome, be?
I wonder if there are other patterns like that.
Being able to tightly define what part of your app is impacted if one of these fail and restore it before the rest of the app is impacted is a game changer.
OTP also deeply force you to think of how you start and how you stop these, taking dependencies into account. It helps in cleaning up stuff.
I guess i'll have to try OTP one time to understand why so many people rave about it. I only know one guy who had made real work with erlang / otp and he told me it was beautiful, but not quite as relevant anymore, because of k8s. But i guess i need to make my own opinion.
With respect to user data, a supervised process can crash and the supervisor can restart with the last known good state of that data. It requires some thought on the part of the developer to achieve this, but it's really powerful.
Plus some 3rd party APIs are rather terrible and if you end up re-sending a request (because your server/container went down after the request got sent to the API, but before the response got back/was processed) then the 3rd party API could end up in an inconsistent state.
Resiliency of a cluster with k8s or similar orchestration, AND resiliency of the individual nodes within that cluster, taking advantage of supervision trees.
(Also, to complete this picture and provide resiliency of the service as a whole, add load balancing across multiple availability zones.)
Restarting an Erlang process and restarting a Docker container is not the same.
Let's say you create an Erlang process per request and that process crash, you don't want to restart the whole node, do you?
Application resiliency and Infrastructure resiliency are not incompatible, in fact, they are complementary.
Also, use libcluster[1] and horde[2], and you will never deal with leader election again. (which Kubernetes does not provide).
I've seen many Kubernetes Operators not handling more than one replicas, and that's a shame if you want High Availability.
[1] - https://hexdocs.pm/libcluster/readme.html
[2] - https://hexdocs.pm/horde/getting_started.htmlWhy though? Why hate k8s so much that you would choose an unreadable language?
I disagree, do they solve the same kind of problem? Yes. But the same problems? No.
If part of your application needs to be restarted, why would you restart the whole application?
Kubernetes is not an excuse to avoid application resiliency. Erlang/Elixir is not an excuse to avoid infrastructure resiliency.
Part of your program may crash, your node may crash. Both needs to be restarted.
It should definitely be possible. With protocols and behaviours it should be easy to build plugins that "just work".
Which goes on to say that there is no silver bullet but there is strength in using Kubernetes, it's kind of a lingua franca.
In my experience with Elixir, bringing everything into the code reduces the surface area of complexity and makes it much easier to model in your head.
Sure it is more powerful and does a lot more things. It is also insanely more complex and hard to use compared to Erlang.
The common idea is: there is a current (possibly wrong) state of the world, and there is a desired state.
Let the system use control feedback to automatically bring itself into a desired state.
i.e.
the current state: I have 3 workers.
the desired state: I want 5 workers.
K8s has a predefined workers topology: deployment -> replica set -> pod -> container. Which uses a small number of patterns (service, StatefullSet, etc.)
While Erlang/OTP allows arbitrary supervision trees composed out of several primitives: one-for-one, one-for-all, etc.
Both have a concept of a cluster composed out of nodes.
K8s does this on the [micro]services level, while Erlang/OTP does it on the clustered monolith level.
K8s has a coarse-grained scheduler, while BEAM has fine-grained.
I've retained the principle even as I've left the Erlang ecosystem because it turns out to be a fairly convenient way of structuring a lot of concurrent network problems. Reifying servers as explicit things in your codebase is really helpful; at that point you might as well have them restart and all that jazz, but to me the real virtue is just the utility of the organizational scheme to my code. That they are composable is also pretty useful.
I am literally taking a brief break just now from taking one server that I was writing that was maintaining a connection to an authorization service and translating that into local actions and splitting it into two servers, one that maintains and manages the connection and one that is solely responsible for the logic of dealing with what's coming in and out from that server, turning what was becoming rather nasty intermingled code into two separated bits of code that each make their own sense. Because the server abstraction is composable, the entire rest of the program doesn't notice; previously I had one "service", and now I've replaced that service with a (sub-)supervisor that starts the two services up. The interface to the rest of the program is identical; the same "start" and "stop" operations are done, and the "tree"-ness is automatic. The rest of the program doesn't care that what used to be one running server is now two cooperating servers because the composability of the concept of "server" hides that away cleanly.
With the servers reified, the process of starting the system is also clean and easy to explain to other people, rather than bespoke processes for starting each one. I've got fairly granular services here compared to some people, but it's still pretty easy to end up with 5-10 distinct types of "servers" in my programs providing internal services. It's nice that they don't all have their own code for handling errors, getting restarted, managing shutdowns, etc.
It is not night and day. I don't want to oversell this, either. It's not like you'd be crazy to not be using this as an organization technique. You can certainly make large programs without this technique. But it is quite convenient to reify the concept of a "manageable tree of services" as explicit objects in your code, and then be able to deal with them as reified objects, just like anything else that we've found convenient to reify as explicit objects over the years rather than leaving implicit, with bespoke unique interfaces for dealing with each individual object. It's one of those things where, individually, each individual such concept isn't a huge deal, but one of the reasons why 2020s programming can be more effective than 1980s programming is precisely that we've done this to some many things and reduced the number of bespoke one-offs we have to deal with. Explicit service trees aren't going to change your life forever but it's one more incremental improvement that's worth adding to the stack.
So genservers are abstracting that,while keeping your code pure
Either the thing you're scaling / respawning is stateless, and then anything from kubernetes to amazon ecs will do.
or it's stateful with a state that you need to persist somehow, and then frankly the hardest part clearly isn't the way web server scale, but rather how the data storage and retrieval mechanism works (along with handling the potential transactionnal behavior requirements).
What am i missing ?