News for hackers. Stuff that matters.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by antirez on Wednesday January 14, 2026 @05:05AM (#46614272)
Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem because of your queue), you have a simple upgrade path. But I bet a lot of Rails apps don't have high volumes, and having to maintain two systems can be just more complexity.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by watercolorblind on Wednesday January 14, 2026 @07:34AM (#46615316)
The primary pain point I see here is if devs lean into transactions such that their job is only created together with the everything else that happened.

Losing that guarantee can make the eventual migration harder, even if that migration is to a different postgres instance than the primary db.

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by phn on Wednesday January 14, 2026 @11:37AM (#46618233)
You can look at it both ways.

Using the database as a queue, you no longer need to setup transaction triggers to fire your tasks, you can have atomic guarantees that the data and the task were created successfully, or nothing was created.

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by byroot on Wednesday January 14, 2026 @04:22PM (#46623694)
That's also something Rails helps abstract away by automatically deferring enqueues to after the transaction completed.

Even SolidQueue behave that way by default.

https://github.com/rails/rails/pull/51426

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by yawboakye on Wednesday January 14, 2026 @05:35AM (#46614426)
the problem i see here is that we end up treating the background job/task processor as part of the production system (e.g. the server that responds to requests, in the case of a web application) instead of a separate standalone thing. rails doesn’t make this distinction clear enough. it’s okay to back your tasks processor with a pg database (e.g. river[0]) but, as you indirectly pointed out, it shouldn’t be the same as the production database. this is why redis was preferred anyways: it was a lightweight database for the task processor to store state, etc. there’s still great arguments in favor of this setup. from what i’ve seen so far, solidqueue doesn’t make this separation.

[0]: https://riverqueue.com/

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by andrewstuart on Wednesday January 14, 2026 @06:48AM (#46614970)
It’s not necessary to separate queue db from application db.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by yawboakye on Wednesday January 14, 2026 @07:29AM (#46615261)
got it. is it necessary, then, to couple queue db with app db? if answer is no then we can’t make a necessity argument here, unfortunately.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by nick__m on Wednesday January 14, 2026 @07:52AM (#46615458)
Frequently you have to couple the transactional state of the queue db and the app db, colocating them is the simplest way to achieve that without resorting to distributed transactions or patterns that involve orchestrated compensation actions.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by yawboakye on Wednesday January 14, 2026 @05:55PM (#46625104)
that’s setting yourself up for trouble, imo. intermediate states solve this problem, and economically. for mature production system see temporal[0]. their magic sauce is good intermediate states.

[0]: https://temporal.io/

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by imtringued on Thursday January 15, 2026 @06:10AM (#46630876)
This feels like an ad because you explained absolutely nothing.

It's one thing to explain to people what a screw driver is and you just happen to sell screw drivers to people who might need them.

It's a wholly different thing to sell the screw driver first and then let people figure out why they need it.

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by yawboakye on Thursday January 15, 2026 @06:59AM (#46631263)
i’m not associated with temporal, nor does the link above have any referrer nonsense in there. i don’t profit from referring to it here. in fact it may well be a household name in the hn community. that out of the way, it’s not wrong to point to a proper resource that can explain and demonstrate my argument better than a couple of words could. temporal is open source[0] so maybe a github link would have been more palatable?

[0]: https://github.com/temporalio/temporal

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by jrochkind1 on Wednesday January 14, 2026 @09:48AM (#46616638)
solid_queue by default prefers you use a different db than app db, and will generate that out of the box (also by default with sqlite3, which, separate discussion) but makes it possible, and fairly smooth, to configure to use the same db.

Personally, I prefer the same db unless I were at a traffic scale where splitting them is necessary for load.

One advantage of same db is you can use db transaction control over enqueing jobs and app logic too, when they are dependent. But that's not the main advantage to me, I don't actually need that. I just prefer the simplicity, and as someone else said above, prefer not having to reconcile app db state with queue state if they are separate and only ONE goes down. Fewer moving parts are better in the apps I work on which are relatively small-scale, often "enterprise", etc.

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by erispoe on Wednesday January 14, 2026 @08:54AM (#46616099)
> it shouldn’t be the same as the production database

Why is that?

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by gregors on Wednesday January 14, 2026 @11:03AM (#46617682)
Here's an example from the circleci incident

https://status.circleci.com/incidents/hr0mm9xmm3x6

and a good analysis by a flicker engineer who ran into similar issues

https://blog.mihasya.com/2015/07/19/thoughts-evoked-by-circl...

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by davidw on Wednesday January 14, 2026 @11:58AM (#46618574)
CircleCI and Flickr are both pretty big systems. There are tons of businesses that will never operate at that scale.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by gregors on Wednesday January 14, 2026 @12:50PM (#46619409)
I don't disagree with that call out. However, we've been through these discussions many times over the years. The solid queue of yesteryear was delayed_job which was originally created by Shopify's CEO.

https://github.com/tobi/delayed_job

Shopify however grew (as many others) and we saw a host of blog posts and talks about moving away from DB queues to Redis, RabbitMQ, Kafka etc. We saw posts about moving from Resque to SideKiq etc. All this to day storing a task queue in the db has always been the naive approach. Engineers absolutely shouldn't be shocked that approach isn't viable at higher workloads.

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by zarzavat on Wednesday January 14, 2026 @09:21AM (#46616353)
If you need to restore the production database do you also want to restore the task database?

If your task is to send an email, do you want to send it again? Probably not.

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by stavros on Wednesday January 14, 2026 @09:30AM (#46616432)
It's not like I'll get a choice between the task database going down and not going down. If my task database goes down, I'm either losing jobs or duplicating jobs, and I have to pick which one I want. Whether the downtime is at the same time as the production database or not is irrelevant.

In fact, I'd rather it did happen at the same time as production, so I don't have to reconcile a bunch of data on top of the tasks.

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by zarzavat on Friday January 16, 2026 @05:49AM (#46645133)
Right, I was referring to logical databases rather than the database server itself.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by stavros on Friday January 16, 2026 @06:05AM (#46645208)
But even for the logical databases, if I want to revert to an earlier state of the database, why wouldn't I want the tasks as well? If I have a bunch of update tasks in flight at that point, wouldn't I want them to actually run? They are a part of the overall state of the system.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by runako on Wednesday January 14, 2026 @09:47AM (#46616635)
SolidQueue uses its own db configuration.

> it shouldn’t be the same as the production database

This is highly dependent on the application (scale, usage, phase of lifecycle, etc.)

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by bgentry on Wednesday January 14, 2026 @01:18PM (#46620004)
Yeah, River generally recommends this pattern as well (River co-author here :)

To get the benefits of transactional enqueueing you generally need to commit the jobs transactionally with other database changes. https://riverqueue.com/docs/transactional-enqueueing

It does not scale forever, and as you grow in throughput and job table size you will probably need to do some tuning to keep things running smoothly. But after the amount of time I've spent in my career tracking down those numerous distributed systems issues arising from a non-transactional queue, I've come to believe this model is the right starting point for the vast majority of applications. That's especially true given how high the performance ceiling is on newer / more modern job queues and hardware relative to where things were 10+ years ago.

If you are lucky enough to grow into the range of many thousands of jobs per second then you can start thinking about putting in all that extra work to build a robust multi-datastore queueing system, or even just move specific high-volume jobs into a dedicated system. Most apps will never hit this point, but if you do you'll have deferred a ton of complexity and pain until it's truly justified.

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by yawboakye on Wednesday January 14, 2026 @05:53PM (#46625075)
state machines to the rescue, ie i think the nature of asynchronous processing requires that we design for good/safe intermediate states.
Re:I Love You, Redis, but I'm Leaving You for SolidQueue by byroot on Wednesday January 14, 2026 @09:40AM (#46616552)
> The ideal situation with Rails would be if there is a simple way to switch back to Redis

That's largely the case.

Rails provide an abstracted API for jobs (Active Job). Of course some application do depend on queue implementation specific features, but for the general case, you just need to update your config to switch over (and of course handle draining the old queue).

Re:I Love You, Redis, but I'm Leaving You for SolidQueue by another_twist on Wednesday January 14, 2026 @03:52PM (#46623168)
Since you're here - https://redis.io/docs/latest/operate/oss_and_stack/managemen...

In AOF mode does Redis write all changes to a WAL ? Is this paired with periodic snapshotting to prevent the log from growing too large ? Does this work in distributed mode or is this single node thing ?