Well, yes and no.
By using Kubernetes, you get a scalable infrastructure. By using OTP/Akka, you get a scalable application.
While there is common problems to both domains, they are still 2 different domains.
For example, using only Kubernetes, you won't have the ability to react to a Pod restart within your application (unless your application is aware of Kubernetes).
Using only OTP/Akka, you still need a workflow for deployment and infrastructure management, and you still need to implement node discover for clustering.
NB: For Elixir, you have libcluster[1] that can use different strategies for node discovery, including using a Kubernetes Service to discover the nodes (as Pods).
EDIT: Using Kubernetes, libcluster, and Horde[2], you get the best of both worlds IMHO.
I find it very useful because "Distributed OTP Application" are a pain IMHO (they must be started during the boot of the BEAM).
[1] - https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...
I also wanted to call out that this ~sentence has just... a bunch of things that seem like jargon/names within the community? As an outsider, I have no idea what they mean:
> make a Genserver or Oban worker and mount it in the supervision tree. The Beam takes care of maintaining the process
it sounds very sci-fi :)
Elixir inherits a library called OTP from erlang which is a set of primitives for building massively concurrent systems.
A Genserver is sort of like a Base Class for a object that runs as an independant process that plugs into OTP. By inheriting/implementing the Genserver behavior, you create an independant process that can be mounted in an OTP supervision tree which runs at the top of your application and monitors everything below it. Out of the box, that means your process can be sent messages by other processes and can send messages itself. If a crash happens, the supervisor will kill it, resurrect it and redirect messages to the new process.
Creating a Genserver is as easy as adding an annotation and implementing a few callbacks.
Genservers are the base on which a lot of other systems build on. Oban, a job worker essentially builds on Genserver to use a postgres table as a job processor. Since its just a Genserver with some added behavior, adding a background worker is as simple as adding a file that inherits from Oban and specifying how many workers for it should be allocated in a config file. The result is that adding a background worker is about as much work for me as adding a controller. No additional work for devops either.
And yes, it is very sci fi. Honestly I'm shocked elixir isn't more widespread. there's very little hype behind it but the engineering is pretty solid. Every scaling bottleneck we've had so far has been in the database (only because we particularly make heavy use of stored procedures)
In the world of OTP (Open telecom platform), a process is the term for what essentially is a green-thread, not an OS process!
So it is: a) much, much more lightweight (IIRC ~ 1Kb) b) scheduled by the Erlang virtual machine (so called BEAM)'s scheduler. The BEAM's schedulers run on a per-thread-basis, inside the BEAM's process c) independently garbage collected, no mutable memory sharing
This isn't a knock against Elixir or the Erlang ecosystem but I would definitely say that Elixir gets a decent amount of hype. Each time a new release comes out it invariably shoots to the front page of HN.
1. if I'm calling directly, I can just use a raw sql query 2. views can be backed with a read only ecto model 3. triggers can be set to run without your ecto code even being aware of it. 4. for custom errors, you can add overides for the error handling in ecto to transform things like deadlocks to 400 errors
https://erlang.org/download/armstrong_thesis_2003.pdf
> supervision tree
See chapter 5 "Programming Fault Tolerant Systems" & section 5.2 "supervision hierarchies"
> genserver / generic server
See chapter 4 "Programming Techniques" section 4.1 "Abstracting out concurrency" and chapter 6 "Building an Application" section 6.2 "Generic server principles"
Genservers are an abstraction encapsulating the typical request/response lifecycle of what we would consider a "server", but applied to a BEAM-specific process. Like "general server".
Oban allows for job processing, instrumented much like you would any other process in the BEAM. This is an external library while genserver is built in.
No OTP, GenServer, etc. Just a webserver and some framework for scheduled jobs, third party or write your own. Config enables individual routes and jobs at the top level. You can deploy one instance with everything enabled, or a hundred instances with the customer-facing web routes enabled in 60, API routes in 20, admin routes in 5, report generation jobs in 10, maintenance jobs in 5, or anywhere in between.
The only discipline you have to stick to is that you must pass information between components in a way which will work when the app is distributed. A shared database is the most obvious, and may be enough. We also used message queues, and at one point one of those distributed caches that were all the rage (Hazelcast / Terracotta / Infinispan / EHCache / etc - anyone remember JavaSpaces and Jini?).
We moved to microservices, despite my love for a monolith with libraries: - to enable different teams to deploy their smaller microservice more easily (without QA, database migrations etc affecting the whole app); - to solve the human temptation of crossing abstraction boundaries.
Not black and white, and both approaches can solve these problems. Open to any feedback!
Really curious about this! What’s the deployment experience? What environment do you use for your production? How do you run/maintain the erlang VM and deploy your service?
One of the big differences we had to make was allocating one node per machine as the beam likes to have access to all resources. in practice this isn't a problem because its internal scheduler is way more efficient and performant than anything managing OS level processes.
That said, a more complex deployemnt story is defiantly one of the downsides. But the good news is that once setup, its pretty damn resilient.
Now of course, our deployment setup is more complex specifically to take advantage of beam such as distributed pubsub and shared memory between the cluster. If you don't need that, you could use dokku or heroku.
As your using postgres have you looked at citus[0] at all?
Obviously, the services will still be necessary where Elixir simply isn't the right tool for the job like machine learning or cryptocurrency.
For elixir:
The whole app is approximately a websocket-based chat app (with some other stuff), and the beauty of OTP + libcluster is that the websocket processes can communicate with each other, whether or not they're running on the same OTP node.
IT does automatically distribute processes across all the cores on a cpu though.
My startup is built out of a single phoenix monolith. Only external dependency is postgres.
Despite this, I've managed to completely avoid all the typical scaling issues of a monolith. Need a service or background worker? I juts make a Genserver or Oban worker and mount it in the supervision tree. The Beam takes care of maintaining the process, pubsub to said service and error handling in the event of a crash.
I'm basicly getting all the best advantages of a monolith
* one codebase to navigate, our team is TINY * all functionality is in libraries
And all the advantages of microservices without the associated costs. (orchestration, management etc)
* bottleneck use cases can live on their own nodes
* publish subscribe to dispatch background jobs
* services that each focus on doing one thing really well
We've accomplished this all through just libraries and a little bit of configuration. At no point so far have we had to..
* setup a message broker (phoenix pubsub has been fine for us. we can send messages between different processes on the server and to the clients themselves over channels)
* setup external clusters for background workers. (Oban was drop in and jsut works. I'm able to add a new worker without having to tell devops anything. the supervision tree allocates the memory for the process and takes care of fault tolerance for me. That leaves me to focus on solving the business problems)
* setup a second github repo. Its a monorepo and at out scale, its fine. We have one library for communicating to the database that every system just uses.
Eventually we'll probably have to start rethinking and building out separate services. But I'm happy that we're already able to get some of teh benefits of a microservice architecture while sticking to what makes monoliths great for mvps. It will be awhile before we need to think about scaling out web service. It just works. Leaves me more time to work on tuning out database to keep up!