Re:Rescuing Resque again (2015)
by AznHisoka on Friday January 15, 2016 @11:57AM
(#10910476)
When i started my project, I used Resque but almost immediately switched to Sidekiq a few months later. I understand it's almost impossible to run into thread safety issues with Resque, but if you're doing anything CPU intensive, Sidekiq really offers much higher performance. The maintainer also makes a living off Sidekiq, selling it to businesses and enterprises too so you can expect more frequent updates.
[ Parent
| on HN ]
> but if you're doing anything CPU intensive, Sidekiq really offers much higher performance
The threading model (LWPs vs. forking) should have no impact whatsoever on how quickly a CPU-intensive task completes.
Unless it's a short-lived CPU intensive task, in which case in matters a great deal.
The overhead of fork() is negligible, well under 0.1ms.
There's also the issue, in ruby, that CoW doesn't really work due to the fact that the GC dirties all pages.
This was fixed in Ruby 2.0.
It matters how many of these tasks can be completed in a fixed amount of time. Creating 50 processes requires more resources than 1 process with 50 threads.
Yeah, Sidekiq is an obvious switch once you find yourself running 20 500MB Rails processes just because you want to be able to handle a few simple tasks concurrently. If you have a few big, long-running, CPU-intensive background jobs, then the benefits are less clear, but if you have lots of small, short running jobs where you're mostly interested in throughput, then Sidekiq makes a lot more sense.
Resque-pool (https://github.com/nevans/resque-pool) -- which every Resque user should be using -- loads your process first, then forks the workers. Forking uses copy-on-write semantics, so as long as you're loading everything before forking, you're conserving a great deal of otherwise wasted memory on what would be shareable pages.