Microservices makes sense, I suppose, but the cases in which distributed services and isolated processes are critical just aren't comment enough in my experience to require sticking specifically to a SOA-like architecture for everything. Especially in a web app like something you'd make in Rails, which already handles exceptions pretty well and rarely require concurrency.

The idea of composing individual applications and programs sounds very powerful, and I like the idea of using it as a way to compose programs written in different languages together (a Rails server talking to a Clojure ML service that uses Rust bindings to do hardcore number crunching, for example). That could be interesting. I don't know what I'd use it for, though.

> Especially in a web app like something you'd make in Rails, which [..] rarely requires concurrency.

What? If you don't need concurrency in a web app that means that your app can only serve one client at a time and other clients have to wait in line. That won't scale to more than a couple concurrent users.

You're right that microservices don't make sense for every web app. Especially when you're still figuring out product-market fit. But once you achieve a certain scale (both in number of users and number of developers) microservices/SOA starts to make a lot more sense.

And that's an area where the Erlang VM really shines. Erlang's concurrency model is awesome and fits really well with the hardware evolution to more cores: lightweight processes, immutability, message passing, location transparency, etc.

I wouldn't be surprised if I've just been insulated from thinking about concurrency with Rails/have never worked on a Rails project that had to think about concurrency in any significant way. Which is a little disappointing, I do want to work on interesting problems instead of being a CRUD monkey forever...

I do think Elixir and Phoenix have a lot of value because Phoenix has the "get shit done" factor of Rails and yet still allows for your application to scale once you've entered the market, and you don't have to drop everything and move to a more scalable language/architecture and slow down. Plus, Elixir's features and syntax make it a lot more accessible than Erlang.

> Especially in a web app like something you'd make in Rails, which already handles exceptions pretty well and rarely require concurrency.

That's interesting considering the reason Phoenix was made is because the author was having a lot of trouble doing concurrency in Ruby.

I think something that confuses the issue here is that while most applications are a great fit for monolithic Rails (or similar), the applications that many of us here seek out to work on are those that are larger (in scale or in functionality) and more complex. I'm not sure how you could check this, but I suspect that people working on bigger applications where SOA thrives are over-represented here. Most of the debate between "you're not gonna need that!" and "I have needed that!" just stems from people attempting to project their own experiences onto everyone.

Consider background jobs using Resque or Sidekiq. Those are a kind of concurrency. These kinds of libraries are unnecessary in Elixir, where you can spin up another process to handle this kind of "background" work outside of the normal request/response flow.

How do you process a million jobs in parallel across many machines? How do you schedule a job to run in a week?

I'm the author of Sidekiq and not trying to debate; I'm simply ignorant of how Erlang, BEAM and the ecosystem solve these common problems, which usually persist no matter which language you use.

(caveat: in practice you'd do this differently, making sure applications were prestarted/preregistered on remote nodes, ensuring your network was already connected and wrapping most of these calls in nicer function calls but for the sake of illustration i've described the naive way to do this)

    %% ping a remote node to connect to it
    pong = net_adm:ping('[email protected]'),
    %% start my application on the remote node
    _ = spawn('[email protected]', application, start, [my_app]),
    %% my_app is a gen_server, an otp process that abstracts
    %% async request/response and makes it look more like
    %% synchronous function calls
    %% this example assumes the gen_server accepts the JobArgs
    %% and immediately returns 'ok' as acknowledgement, but does
    %% the work asynchronously
    ok = gen_server:call({my_app, '[email protected]'}, {do_some_work, JobArgs})

if you wanted to distribute a number of jobs over a number of nodes you'd just repeat this process for each node and distribute your jobs however you choose, either by round robin or something more complicated like rendezvous hashing or something

this example assumes you're uninterested in the responses, but if you wanted to receive the results you could either send a pid (process identifier) along with the request that would be setup to collect the results and do any further processing or you could spawn a local process that made the actual call and blocked until it received the results

as for scheduling a job a week in the future, the naive way would be to setup a timer that, when it expires, spawns a process that can execute an arbitrary function. no one would do this, normally, however. you'd probably want to use something like sidekiq that can store the job off node and poll regularly to see if there's any jobs you should run

For your first question, I use the toniq library: https://github.com/joakimk/toniq. As biokoda mentioned, there's nothing stopping you from doing this by literally just spawning a ton of processes, but you do get a lot of nice things out of the box with toniq or another task queue framework. For example, toniq gives you automatic task retrying with exponential backoff, a max-concurrency option, and task backup with redis.

It doesn't have a scheduled job API yet, but there is a note in the readme that explains how to do this with quantum-elixir: https://github.com/joakimk/toniq#how-do-i-run-scheduled-or-r....

> How do you process a million jobs in parallel across many machines?

Start the processes that execute the code, get notified when they are finished. Literally a couple of lines of Erlang/Elixir. Not some large framework that does all that for you, it's in the language. It is built to run a lot of tasks in parallel and across many machines.

> How do you schedule a job to run in a week?

Like anyone else, put it in some permanent storage like redis. You can also keep it in memory, you can also use mnesia which is a distributed database that comes with Erlang.

Let's go for that one. Processing a million jobs in parallel is not hard

  my_list_of_job
  |> distribute_tasks() 
  |> Enum.map(fn x -> Task.Supervisor.async(x, my_list_of_job) end)
  |> Enum.map(fn x -> Task.await(x, your_timeout) end)

That is will distribute your worload on all the nodes you have that implement a runner for that. Of course you may want to make things a bit different, and i did not covered your load balancing that i hidden in distribute task, nor any error handling, but that would not be far harder imho. Until you enter the partition and idempotent part, but even that would not be that hard.

There is a long running project to work on that to produce a nice job queue lib on Elixir Slack but noone on the interested group have time for now.

To schedule a job to run in a week you have a couple way. An easy first approach is to use Process.send_after/3 to send yourself a message after a week and handle that message in a week. Of course if you want persistence, etc, you will have to do it a bit differently, but not so much....

edit : now use the fn form instead of the shorthand & for ease of comprehension for non seasoned Elixir devs

That's a really good question and I'd be delighted if somebody answers it adequately.

I heard that Erlang/OTP (and thus Elixir) can transparently distribute work no matter where it gets physically executed but I'd just love to see an example -- freely accessible code, demo site, everything.

I don't think those two exclude each other, I intend to upgrade my Go skills, but honestly Elixir is preferred.

Something that everybody seems to be missing is that a microservice can serve as an endpoint for all type of clients: your web app, a rich browser client, your ios app, your android app, your smart tv app, third parties, your reporting application, etc.