Re:Lovely Week with Elixir
by moreaccountspls on Wednesday May 20, 2020 @05:18PM
(#23251913)
Slightly offtopic, but consider a pattern of persisting the state machine of a given task instead of the task queue itself. For example, for a daily email job, you might have three states: UNSENT, BEGIN(begin_time), SENT. Then you can just have a supervisor job that scans your table or whatever and schedules jobs, retries, etc. as needed. Now you don't have to worry about the queue state at all!
Having a queue means you have a distributed system. How do you handle network problems, errors/retries, back pressure? OTP has excellent idiomatic tools for all that and more.
Elixir developers who need a queue generally reach for Rabbit (which any language can use), or something backed in a database like rihanna[1] or honeydew[2]. Rolling your own distributed system is very much a last resort, and despite its excellent concurrency characteristics the BEAM still lacks basics such as a battle-tested raft implementation.
[1]https://github.com/samsondav/rihanna [2]https://github.com/koudelka/honeydew
> and despite its excellent concurrency characteristics the BEAM still lacks basics such as a battle-tested raft implementation.
A couple of years ago the RabbitMQ team has published a raft library[1] which they use in their implementation of persistent queues. It has a flexible API and implementing your own state machines is quite straightforward, as it follows the OTP gen_* behaviour paradigm.
And by now, I'd say it's pretty well battle-tested.
For any serious application you want a job like that to be persisted so you can guarantee it runs even if your application is restarted.
I know that Erlang/Elixir is designed for stateful applications and if you have a cluster and do hot deploys this is less of a problem, but who does that? Most Elixir applications I've seen are deployed as stateless systems just like any Ruby, Python, Node etc systems.