News for hackers. Stuff that matters.
Re:Show HN: Sneakers – Fast background processing for Ruby by bdcravens on Tuesday January 07, 2014 @01:17PM (#7018793)
Look forward to checking this out. The biggest win regarding Sidekiq for me wasn't so much performance (which is was definitely better than Resque) but it's retry logic, error reporting, batch handling (granted, that's paid), ease of plugging into an existing app's security model, and more.

Without having looked at it yet, how does the pre-built UI compare? From an initial glance, it looks like anything that's missing is available via the DSL, but Sidekiq gave me so much of what I needed out of the box (having come from a custom-made solution using SQS + open source CFML) it made my head explode.

Re:Show HN: Sneakers – Fast background processing for Ruby by jondot on Tuesday January 07, 2014 @01:44PM (#7018969)
I can definitely say that RabbitMQ (and the awesome bunny ruby library) is doing the heavy lifting.

Batch handling - is actually "prefetch" in RabbitMQ/AMQP. Error reporting - via logging, and a "dead letter mailbox" (called a dead-letter-exchange in RabbitMQ) which is a great enterprise integration pattern for properly handling errors and retries in jobs.

RabbitMQ does the heavy lifting in another surprising facet - the UI. The management UI is excellent and at some point in time I decided not to compete with it by creating my own UI (you have to pick your fights :).

Here's some more info about the management plugin for RabbitMQ.

http://www.rabbitmq.com/management.html

Re:Show HN: Sneakers – Fast background processing for Ruby by bdcravens on Tuesday January 07, 2014 @02:15PM (#7019189)
The thing about Sidekiq is the management is closer to my app, but I'd of course need to look into Rabbit's UI before making any decisions.

I suspect that if this project takes off someone will put together a front end that mirrors Sidekiq's in short order.

Re:Show HN: Sneakers – Fast background processing for Ruby by chadcf on Tuesday January 07, 2014 @01:22PM (#7018827)
Sidekiq's retry and robust handling of jobs (at least with reliable queuing in pro) was the big win indeed. I had a app I tried switching over to it though and it did not go well. In my case I'm running background jobs that take 4-5 minutes to complete and are pretty cpu intensive, and I suspect because sidekiq is threaded it's only using one CPU core and completely thrashing the machine. I can run about twice as many workers with resque as I can with sidekiq, so I had to switch back. Sadly now I've lost the reliable queuing and automatic retries (the latter at least is not too hard to implement in an ensure block).

I do wonder how this one would work out though, will have to give it a try at some point.

Re:Show HN: Sneakers – Fast background processing for Ruby by jondot on Tuesday January 07, 2014 @01:49PM (#7019013)
More or less the same. Plus it will use all cores on MRI, so you can keep using gems with C-Extensions (i.e. you don't have to run rbx or JRuby to max all cores).

Discussing reliability - this is something that sadly Sidekiq will never give you, by virtue of the fact that it uses Redis. RabbitMQ can be clustered in active-active mode, which means you have won over reliability here by just using a cluster.

When comparing queue systems, comparing Sidekiq+Redis to RabbitMQ is a bit unfair - because RabbitMQ was born to do this. And that's why if you're doing proper background jobs and messaging it's better to pick the right tool.

That being said, I do keep using Sidekiq for small Rails apps for the typical background emailers, denormalizers, etc. But I keep an eye open for when I realize that I'm doing proper messaging - in which case I'll switch over to something like Sneakers.

Re:Show HN: Sneakers – Fast background processing for Ruby by MoOmer on Tuesday January 07, 2014 @02:35PM (#7019330)
I've got to say that the more I hear "Redis can never be reliable" the more I cringe. It just seems like one of those things that's been said and repeated without people stopping to fact-check along the way.

Redis Clustering tutorial: http://redis.io/topics/cluster-tutorial

Redis persistence (using AOF or RDB or both): http://redis.io/topics/persistence

Re:Show HN: Sneakers – Fast background processing for Ruby by jondot on Tuesday January 07, 2014 @03:07PM (#7019550)
Right now Redis cannot be clustered production-ready. I wish. As I stated in the Wiki, you'll have to pry Redis from my dead body, I am very happy with it, and for me its a true swiss army knife and I've used it as such.

Even though it doesn't have clustering - it's rock solid in production and I haven't experienced a drop in one of my Redis servers in around 3 years.

That being said, if you are building a system where reliability is an explicit requirement you can't take those risks.

Re:Show HN: Sneakers – Fast background processing for Ruby by druiid on Tuesday January 07, 2014 @02:43PM (#7019381)
Still in beta/alpha as far as I last checked (we've been waiting for it). Additionally as far as clustering is concerned there still seem to be a lot of 'unknowns' for me with clustering Redis. Using a cluster in RabbitMQ is dead simple and it just works.
Re:Show HN: Sneakers – Fast background processing for Ruby by milkshakes on Tuesday January 07, 2014 @03:56PM (#7019853)
you might want to do some fact checking yourself before you cringe.

i would start here: http://aphyr.com/tags/Redis

Re:Show HN: Sneakers – Fast background processing for Ruby by bdcravens on Tuesday January 07, 2014 @01:34PM (#7018896)
On a long-running job, I can see that being an issue.

However, what was your concurrency level set at? I know when I've set it to one, I've had good success with the most demanding tasks. You essentially lose the threaded benefit, but keep the other benefits.

Re:Show HN: Sneakers – Fast background processing for Ruby by djur on Tuesday January 07, 2014 @01:27PM (#7018854)
What Ruby version/impl are you using? Native threads in a non-GIL implementation should use all available cores.
Re:Show HN: Sneakers – Fast background processing for Ruby by steveklabnik on Tuesday January 07, 2014 @07:23PM (#7020957)
Resque maintainer here: in recent versions we do the RPOPLPUSH stuff, so 'reliable queueing' should be there.

At least, as reliable as you can get with Redis...