We're currently using ecto_job which works really well for us, so we have no reason to switch. Plus we like it's in different tables.
PG is an amazing tool that can handle more than most people think (or at least more than I thought).
What exactly is fragile about this approach? We've supported scheduled tasks in TaskTiger (https://github.com/closeio/tasktiger) via Redis's sorted sets and haven't had any issues.
Using sorted sets for scheduled tasks isn't the fragile part, gluing it all together to prevent losing jobs by shifting into backup lists (or hashes) is the fragile part in my experience.
That definitely does the job. My point is that it is much more complex than a select/update clause in SQL.
[0] https://github.com/closeio/tasktiger/blob/master/tasktiger/r...
Stream does solve the backup queue issue, but not being able to filter messages from the consumer's pending list makes implementing retry mechanism and exponential backoff hard (without reaching to sorted set).
Though, I can understand the Redis team's decision to keep it simple.
After replicating most of Sidekiq’s pro and enterprise behavior using older data structures I attempted to migrate to streams. What I discovered is that all the features I really wanted were available in SQL (specifically PostgreSQL). I’m not the first person to discover this, but it was such a refreshing change.
That led me to develop a Postgres based job professor in Elixir: https://github.com/sorentwo/oban
All the goodies only possible by gluing Redis structures together through lua scripts were much more straightforward in an RDBMS. Who knows, maybe the recent port of disque to a plug-in will change things.