Is it though? At least in my line of work I don't think I've ever run into this. I feel like I've always been able to distribute just fine with workers/queues. If I even suspected it would I'd look into it more, but generally I find distributing across systems to be a software architecture-level and not language-level work; perhaps I'm missing something, however.
The other part is that you can build more efficient systems by relying on this. If you have a machine with 8 cores, it is more efficient to start a single process that can leverage all 8 cores and multiplex on both IO and CPU accordingly. This impacts everything from database utilization, to metrics, third-party APIs, and so on.
The Phoenix web framework also has great examples of using distribution to provide features like distributed pubsub for messaging and presence without external dependencies.
However, when it comes building systems, then I agree with you and I would probably use a queue, because you get other properties from queues such as persistence and making the systems language agnostic.
I hope this clarifies it a bit!
For caching you have SO many options which are already built in, ets, agent, ets + gen_server, or even reaching out for a library like nebulex.
Another example is recurring jobs, you can create a gen_server that will run a job every x hours in roughly 50-100 lines of code depending on how complex your problem is.
I rarely feel the need to reach out for external dependencies. Not that there is anything wrong with that, it's just that you now have a wider array of tools to work with and some problems are just solvable with a couple of lines of code instead of having to reach out for external tooling.
I recently built a distributed work queue, generally in Ruby I would use something like sidekiq, however elixir made me feel like "hey you can write your own" which generally is not recommended since you don't want to re-invent the wheel, but if you are doing something that breaks away from the existing solution, having the ability to craft a custom solution that works for your specific set of problems is extremely powerful, you can get much more creative, and the important thing is it got done fast (I wrote a distributed job scheduler in 2 weeks + 1 week to clean it up and work out the kinks, it's already running stably in production)
Workers/queues in languages like Ruby have problems like,
* Require very specific ergonomics(for example, don't hand the model over, hand over the ID so you can pull over the freshest version and not overwrite)
* They require a separate storage system, like your DB, Redis, etc. This doesn't sound big, but when doing complex things it can turn into hell.
* They have to be run in a separate process, which makes deployment more difficult.
* They're slow. Almost all of them work on polling the receiving tables for work, which means you've got a lag time of 1-5 seconds per job. Furthermore, the worse your system load, the slower they go.
* You can't reliably "resume" from going multi-process. Lets say you're fine with the user waiting 2-3 seconds to have a request finish. With workers/queues, you either have to poll to figure out when something finished(which is not only very slow, but error prone), or you have to just go slow and not multi-process, making it into a 8-10 second request even though you've got the processing power to go faster.
So, you've got all that. Or in Elixir, for a simple case, you replace `Enum`(your generic collection functions) with `Flow` and suddenly the whole thing is parallel. I mean that pretty literally too- when I need free performance on collections, that's usually what I do. Works 95% of the time, and that other 5% is where you need really specific functionality anyway, and for those, Elixir still has the best solution to it I've ever seen.
Shopify, for example, use Resque (Ruby + Redis) to process thousands of background jobs per second.
> * Require very specific ergonomics(for example, don't hand the model over, hand over the ID so you can pull over the freshest version and not overwrite)
This is good practice but certainly not a requirement. You can pass objects in a serialized format like JSON or use Protobuf etc.
> * They require a separate storage system, like your DB, Redis, etc. This doesn't sound big, but when doing complex things it can turn into hell.
ETS and Mnesia aren't production ready job queues, unfortunately: https://news.ycombinator.com/item?id=9828608
> * They have to be run in a separate process, which makes deployment more difficult.
Background tasks have different requirements so this is a good idea regardless.
> * They're slow. Almost all of them work on polling the receiving tables for work, which means you've got a lag time of 1-5 seconds per job. Furthermore, the worse your system load, the slower they go.
Redis queues have millisecond latency and there's no polling. Resque and Sidekiq use the BRPOP to wait for jobs. BRPOP is O(1), so it doesn't slow down as the queue backs up.
PG has LISTEN/NOTIFY to announce new jobs or the state change of an existing job so there's no need to poll. SKIP LOCKED also prevents performance degrading under load.
> * You can't reliably "resume" from going multi-process. Lets say you're fine with the user waiting 2-3 seconds to have a request finish. With workers/queues, you either have to poll to figure out when something finished(which is not only very slow, but error prone), or you have to just go slow and not multi-process, making it into a 8-10 second request even though you've got the processing power to go faster.
There are multiple other options here which are better:
Threads - GIL allows parallel IO anyway and JRuby has no GIL
Pub/Sub - Both Redis and PG have a great basic implementation usable from the Ruby clients
Websockets - Respond early and notify directly from the background jobs
ie, requiring very specific ergonomics. If you have to change what you're doing, it's a new domain to learn.
> ETS and Mnesia aren't production ready job queues, unfortunately: https://news.ycombinator.com/item?id=9828608
I didn't mention ETS or Mnesia? The OP was talking specifically about using job queues to get concurrency/parallelism, in which case you absolutely don't need job queues. If you need a job queue, you need a job queue.
> Background tasks have different requirements so this is a good idea regardless.
Why? You're just stating this like it's obviously true, and honestly I can't think of a time I significantly wanted a different system doing my jobs than the one handling requests.
> Redis queues have millisecond latency and there's no polling. Resque and Sidekiq use the BRPOP to wait for jobs. BRPOP is O(1), so it doesn't slow down as the queue backs up.
Redis queues have millisecond latency, Ruby using Redis queues does not. That's the part that polls when there's nothing else going on. If you're never running out of jobs to do then your latency is fast, but you're also not accomplishing things as fast as possible(since it's waiting on whatever is in front of it).
If this isn't true anymore, then alright, but last I used Sidekiq(early 2018), the latency to start processing a job was often greater than a second.
> Threads - GIL allows parallel IO anyway and JRuby has no GIL
And are incredibly difficult to use and pass information back and forth(hence why Elixir exists at all- Jose Valim was the person implementing this on the Rails core team).
> Pub/Sub - Both Redis and PG have a great basic implementation usable from the Ruby clients
Can certainly work, to be honest I never tried this because of the complexity of initial setup and how green I was when I needed it.
> Websockets - Respond early and notify directly from the background jobs
Which Ruby has a lot of trouble maintaining performantly. When my original team went to use Rails5 sockets, we found we could barely support 50 sockets per machine.
---
It's worth saying, I'm not saying one shouldn't use Ruby- the place I work right now is a primarily Ruby shop, and my Elixir work is for event processing and systems needing microsecond response times. But, we've also built things in Elixir that normally I would use Ruby or JS for, and not only does it do well, but often it's write it and forget it, with deployment being literally "run a container and set the address in connected apps".
Resque & Sidekiq build this in by converting job arguments to JSON. There's nothing extra to learn.
> I didn't mention ETS or Mnesia? The OP was talking specifically about using job queues to get concurrency/parallelism, in which case you absolutely don't need job queues. If you need a job queue, you need a job queue.
Sorry, I thought you were talking about building a background job system in Erlang using out of the box OTP but it sounds like you're actually talking about trying to get parallelism in Ruby by doing RPC over Sidekiq? That's always a bad idea!
> Redis queues have millisecond latency, Ruby using Redis queues does not. That's the part that polls when there's nothing else going on. If you're never running out of jobs to do then your latency is fast, but you're also not accomplishing things as fast as possible(since it's waiting on whatever is in front of it).
Ahhh! When Mike Perham says "Sidekiq Pro cannot reliably handle multiple queues without polling" what this really means is a Redis client can only block on and immediately process from the highest priority queue. The lower priority queues are only checked when blocking timeout expires. There's no "check all queues and sleep" polling loop which adds artificial latency.
> And are incredibly difficult to use and pass information back and forth(hence why Elixir exists at all- Jose Valim was the person implementing this on the Rails core team).
Jose Valim didn't join Rails core until a couple years after Josh Peek (now working for GitHub) made Rails thread-safe.
> And are incredibly difficult to use and pass information back and forth(hence why Elixir exists at all- Jose Valim was the person implementing this on the Rails core team).
It's really not that hard anymore!
results = ['url1','url2'].parallel_map{|url| HTTParty.get(url) }
2012-2013 onwards Ruby got great libraries like concurrent-ruby and parallel that make things a lot easier.
> Which Ruby has a lot of trouble maintaining performantly. When my original team went to use Rails5 sockets, we found we could barely support 50 sockets per machine.
ActionCable is designed for convenience not performance. https://github.com/websocket-rails/websocket-rails will handle thousands of connections per process.
With kubernetes you have an endpoint per service that route and load balance to the correct machine.
It seem to me you already got all the benefits from using actor but with less lock-in to a language.
Not to mention, microservices are not always the answer. And even if they were, they're still an insane amount of more work than literally changing what functions you call.
I'm not saying you should never use an RPC, but I've significantly reduced the times I'd want to use one. The only reason I even advocate for one now, is because I prefer to empower developers to use whatever language makes them happy, even if I personally greatly prefer Elixir.
It’s really not a distributed monolith as each node can deploy with different code. More importantly actors and supervision trees really help keep projects organized. It’s easy to use PubSub mechanisms or just named actors to communicate between services.
As an example I recently took a project that ran on a single IoT device and moved a chunk of it that managed a piece of hardware to another IoT device connected by Ethernet. It only took moving a handful of files and renaming a few modules. It took longer to figure out why multicast wasn’t working than to refactor the app. There are some limitations with type specs not working as well as I’d like with PubSub style messages (most type checking is done via api functions, not on individual messages).
You are joking right? You mean web developers?
As for distribution, again, this isn't necessarily exclusive, but the right primatives are there and work well to start with. You could have good reasons for a bigger separation between nodes as well.
Erlang has some warts too, of course. For me, the warts are usually about scale, oddly enough. BEAM itself scales very well, but some parts of the OTP don't, often because of the difference in expectations between a telcom environment and a large scale internet service. Two examples:
A) the (OTP) tls session cache is serviced by a single process and in earlier versions, the schema and queries were poorly designed so you could store multiple entries for a single destination, and the query would retrieve all and then discard all but the first. When you were making many connections to a host that issued sessions but didn't resume them, all of the extra data could overwhelm that one process, and resulting in timeouts attempting to connect to any tls host. This was fixed in a release after r18, I believe, to store only one session per cache key, and the cache was plugable before then, but it wasn't fun to find this out in production.
B) reloading /etc/hosts and querying the table it loads into weren't done in an atomic way. I believe this is fixed in upstream as well, but queries satisfied by /etc/hosts were actually two queries on the same table, and reloading the table was done by clearing and then loading, so the second query could fail unexpectedly. This led to the bundled http client getting stuck, despite timeouts.
Debugging __any__ distributed system is difficult, this is why monitoring and tracing should be first class citizens in your deployments. It seems they are not for you.
One of the alternative languages you mention is single-threaded, and the other has a global interpreter lock (in its most common implementation). That Elixir is superior to them for parallel programming doesn't really say much.
In elixir you really get the full power of multi core and support for distributed computing out of the box.
Code that would have been beyond my pay grade or wouldn’t even imagine to write in Ruby or JavaScript is now easily reasoned about and maintained in projects. I can write succinct code that is easy to read, is fast, able to take advantage of multiple cores, less error prone, which I can scale to multiple machines easily.
The erlang scheduler is so damn powerful and it feels amazing to be able to execute your code on multiple machines with a simple distributed task which is built in as a standard functionality of the language.
I’ll end this note saying that, look at the problem you are trying to solve. If you need multi core and distributed features (which is generally more common than you think) elixir is truly your friend.
I can say without a shadow of a doubt the project I’m building right now would not be progressing as fast as it is if I picked anything other than Elixir. You get a lot of bang for your buck when it comes to productivity in the domain that elixir solves for.