News for hackers. Stuff that matters.
Re:Rescuing Resque again (2015) by thibaut_barrere on Friday January 15, 2016 @02:55PM (#10911670)
In my opinion, Resque so far failed because like a lot of OSS projects (and unlike Sidekiq), it isn't built on sustainable economical grounds - from the maintainer POV.

Bringing outside "funding" can work for a while (see JRuby), but when the funding stops, it only underlines that the financial engine should really be carefully "designed" at the core of the OSS project, for things to be maintainable on the very long run. Otherwise history is bound to repeat itself.

Large projects can enjoy a large pool of contributors, sometimes with less risks of burn-out (often seen on small OSS projects around me at least), each contributing marginally, with a few paid core members (either directly or indirectly via by-products of a well paid activity), but for smaller projects like Resque, it's more likely a single individual full-time, paid by the project.

Again Sidekiq is really a great inspiration here for me (disclaimer: I plan to launch a "pro" offering for an OSS project I manage at http://kiba-etl.org/, and maintained another open-source for a while before which ultimately went to the cesspit).

Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @02:57PM (#10911687)
Yup, this is exactly it. The only reason I was able to do all the open source work I did was because of my job at Jumpstart Labs, and once that ended, well...

That said, as you mentioned, this is pretty much the default state for the large, large part of open source projects. Something's got to give somewhere, eventually.

Re:Rescuing Resque again (2015) by thibaut_barrere on Friday January 15, 2016 @03:05PM (#10911723)
What is your take on the Sidekiq model? I'm really curious to hear more about that :-)

I see a number of people around me currently thinking about putting a "fair salary" at the core of their future open-source work. Often it completely changes what the delivered value should ultimately be etc (something I think Mike did think about - I believe).

Do you have thoughts on that approach?

Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @03:09PM (#10911755)
My thoughts are mostly "I'm glad it's working out for Mike, and I hope others can follow his path." I personally have taken pursued the patron model, but that also is because with Rust, that's the only real way to do it.
Re:Rescuing Resque again (2015) by thibaut_barrere on Friday January 15, 2016 @03:16PM (#10911802)
By patron model, to clarify, you mean a single large company paying most of the bills and bringing funding, right? (either because they want to promote innovation, or this generates financial gains, etc etc)?

I would also think that for a language, this is probably the best model (even if it failed with IronRuby, or if JRuby funding sources moved over time etc).

This is a fairly large endeavour, so it makes sense (at least I believe it!).

Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @03:20PM (#10911826)
Yes. Though I mean me personally, not the language itself. I wanted to work on Rust full time, Mozilla wanted to hire someone to work on Rust full time, it fits.
Re:Rescuing Resque again (2015) by thibaut_barrere on Friday January 15, 2016 @06:39PM (#10913161)
It's great that it works for you one on one, sincerely! I hope though that we will see more occurrences of OSS projects where N customers are sending money to directly finance the sustainability of the OSS product, just because it delivers value for them (not a moral obligation or PR etc - note that I'm not stating that Mozilla is doing this!).

I feel for libraries & components (unlike languages maybe, as we said), at that specific scale, it can work very well.

Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @06:40PM (#10913169)
I do too.
Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @02:04PM (#10911340)
Hi all! Happy to answer any questions about this effort. I'm targeting 1.26.0 sometime in the next couple of weeks.
Re:Rescuing Resque again (2015) by sanderjd on Friday January 15, 2016 @02:45PM (#10911600)
Thanks for all your hard work on this over the years!

Something I've wondered for awhile: what are the benefits of continuing Resque development, rather than focusing on making the migration path to Sidekiq as painless as possible? Put another way, what do you hear from users of Resque (like Shopify, as you mentioned in another comment) as the reason that they don't want to migrate off of it?

I was curious about this way back in 2012 when you announced work on Resque 2.0, and I had already made the switch to Sidekiq. I thought there must be a good reason to keep working on it, but I wasn't sure what it was, and I'm still not sure, so I figured I'd just ask.

Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @02:51PM (#10911639)
Sidekiq has a lot of attention amongst new developers, but has at least an order of magnitude less users, at least, if gem downloads are something to go by. So from my standpoint, those people deserve bugfixes, and "throw away what works just to transition to something new" isn't really a great response.

Threading vs process-based jobs is a tradeoff. Threads are not universally better.

Re:Rescuing Resque again (2015) by mperham on Friday January 15, 2016 @10:50PM (#10914156)
As of today, Resque has 6,308,611 downloads. Sidekiq has 5,669,265 downloads. Can you explain why you think "order of magnitude"?
Re:Rescuing Resque again (2015) by steveklabnik on Saturday January 16, 2016 @09:44AM (#10915366)
Hey Mike! Sorry, I 1) mis-read that number, 2) when we used to release more often, the download counts for latest versions used to be 10x yours. I don't think that's true at this point. I will update that cache in my brain and not say this again, I apologize.
Re:Rescuing Resque again (2015) by mperham on Saturday January 16, 2016 @09:49AM (#10915376)
No worries, I just wanted to make sure you didn't have access to super secret Ruby usage numbers that I didn't!
Re:Rescuing Resque again (2015) by steveklabnik on Saturday January 16, 2016 @10:09AM (#10915434)
I wish. :)
Re:Rescuing Resque again (2015) by sanderjd on Friday January 15, 2016 @03:01PM (#10911706)
Thanks for the quick answer. Makes sense.
Re:Rescuing Resque again (2015) by jweir on Friday January 15, 2016 @02:51PM (#10911643)
Is Github still using Resque?
Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @02:55PM (#10911676)
They are as far as I know, though they've been on a fork for a while, due to https://github.com/resque/resque/issues/552

One employee has mentioned that they're willing to work on it: https://github.com/resque/resque/issues/1372#issuecomment-17...

Re:Rescuing Resque again (2015) by avitzurel on Friday January 15, 2016 @02:40PM (#10911577)
I have to be honest here. My initial thought when reading this a couple of weeks back was that some things are better left to dead.

Resque has been an example of how not to run an open source project for a while now. (3+ years).

We were working with Resque for a long while and had a lot of issues with performance.

Sidekiq is awesome and it's being well managed and releases are frequent.

But, after thinking about it, I think that Resque has it's place in the community, I definitely think that more than a single project doing this task is needed.

Many people are already familiar with Resque and trust it, those people should be taken care of.

I wish everyone involved a lot of luck with it.

Re:Rescuing Resque again (2015) by sanderjd on Friday January 15, 2016 @02:52PM (#10911646)
This is another good answer to the question[0] I just posed to Steve. You're right that competition is generally good. On the other hand, it would (arguably?) be much more "Rails-y" to have a single conventional solution that works well and gets a lot of eyeballs. Neither Resque nor Sidekiq are that thing though. It has always struck me as a bit odd that Rails outsources the decision on this component, which nearly every Rails project needs. I think previously it was probably to avoid an external dependency (on Redis), but perhaps that could be reconsidered now that >5.0 has eschewed that avoidance.

Having said all that, one of the things that is most exciting to me about Elixir/Phoenix is that this sort of thing doesn't even come up, because a great and mature solution is built directly into the runtime.

[0]: https://news.ycombinator.com/item?id=10911600

Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @02:54PM (#10911659)
Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @02:53PM (#10911651)
Thanks!

  > Resque has been an example of how not to run an open source project
  > for a while now. (3+ years).
Resque is the story of a gem that _everyone_ uses, yet nobody invests in. So the maintainers burn out, and burn out, and burn out. Chris, Terence, Me... trying not to do that again ;)
Re:Rescuing Resque again (2015) by avitzurel on Friday January 15, 2016 @05:16PM (#10912629)
The problem actually started before you guys came to the Resque :)

I remember you commented on my open source project (Resque plugin), I was impressed with the reach of how deep you want to go with this.

I agree with what you said 100%, everyone is using it and no one invests in it. That's why it's hard to maintain and that's why Sidekiq is "winning". Since money is a justified investment to any company for that project and it keeps the project going.

Re:Rescuing Resque again (2015) by AznHisoka on Friday January 15, 2016 @11:57AM (#10910476)
When i started my project, I used Resque but almost immediately switched to Sidekiq a few months later. I understand it's almost impossible to run into thread safety issues with Resque, but if you're doing anything CPU intensive, Sidekiq really offers much higher performance. The maintainer also makes a living off Sidekiq, selling it to businesses and enterprises too so you can expect more frequent updates.
Re:Rescuing Resque again (2015) by otterley on Friday January 15, 2016 @12:04PM (#10910524)
> but if you're doing anything CPU intensive, Sidekiq really offers much higher performance

The threading model (LWPs vs. forking) should have no impact whatsoever on how quickly a CPU-intensive task completes.

Re:Rescuing Resque again (2015) by andrewvc on Friday January 15, 2016 @12:25PM (#10910685)
Unless it's a short-lived CPU intensive task, in which case in matters a great deal.
Re:Rescuing Resque again (2015) by otterley on Friday January 15, 2016 @06:07PM (#10912962)
The overhead of fork() is negligible, well under 0.1ms.
Re:Rescuing Resque again (2015) by andrewvc on Friday January 15, 2016 @10:31PM (#10914101)
There's also the issue, in ruby, that CoW doesn't really work due to the fact that the GC dirties all pages.
Re:Rescuing Resque again (2015) by mperham on Friday January 15, 2016 @10:46PM (#10914144)
This was fixed in Ruby 2.0.
Re:Rescuing Resque again (2015) by AznHisoka on Friday January 15, 2016 @01:40PM (#10911174)
It matters how many of these tasks can be completed in a fixed amount of time. Creating 50 processes requires more resources than 1 process with 50 threads.
Re:Rescuing Resque again (2015) by sanderjd on Friday January 15, 2016 @02:37PM (#10911552)
Yeah, Sidekiq is an obvious switch once you find yourself running 20 500MB Rails processes just because you want to be able to handle a few simple tasks concurrently. If you have a few big, long-running, CPU-intensive background jobs, then the benefits are less clear, but if you have lots of small, short running jobs where you're mostly interested in throughput, then Sidekiq makes a lot more sense.
Re:Rescuing Resque again (2015) by otterley on Friday January 15, 2016 @06:05PM (#10912953)
Resque-pool (https://github.com/nevans/resque-pool) -- which every Resque user should be using -- loads your process first, then forks the workers. Forking uses copy-on-write semantics, so as long as you're loading everything before forking, you're conserving a great deal of otherwise wasted memory on what would be shareable pages.
Re:Rescuing Resque again (2015) by fphilipe on Friday January 15, 2016 @12:40PM (#10910762)
The "again" in the title confused me. I knew that I read an article along these lines a while back so I thought this is yet another one. Turns out it was this post, which is dated October 23, 2015.
Re:Rescuing Resque again (2015) by polysaturate on Friday January 15, 2016 @12:43PM (#10910774)
Aw poo...missed the date when I read it, and haven't seen it before. I'll update the title.
Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @02:05PM (#10911345)
It's again because it's calling back to this one from 2012! http://blog.steveklabnik.com/posts/2012-09-22-resque--let-s-...
Re:Rescuing Resque again (2015) by whatupdave on Friday January 15, 2016 @01:34PM (#10911127)
I almost alway use Que [1] now. It uses postgres advisory locks instead of Redis so one less dependency to worry about. You probably only need Redis if you're running a tonne of jobs.

[1] https://github.com/chanks/que

Re:Rescuing Resque again (2015) by bdcravens on Friday January 15, 2016 @01:45PM (#10911217)
If you're already using Resque (the audience for the blog post), you're already using Redis, so Sidekiq would make more sense. Additionally, a quick glance at the syntax for a que job appears that you'd need to change more code (I could be wrong), and the available UI seems to lag Sidekiq's.

Reading this: "Que's job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn't able to clean it up. The most common cause of this is long-running transactions, so it's recommended to try to keep all transactions against the database housing Que's job table as short as possible." I wonder what consistutes "high load". I throw hundreds of thousands of jobs atop Sidekiq often; if you're using the same database to manage your jobs, it seems that would dogpile your database (since each of those jobs is likely making db calls, and if it's a Rails app, you're typically using the same database)

Re:Rescuing Resque again (2015) by callmeed on Friday January 15, 2016 @03:02PM (#10911711)
Rails 5 will require Redis so in the future it will be a dependency anyway ...
Re:Rescuing Resque again (2015) by djur on Friday January 15, 2016 @06:17PM (#10913024)
Rails 5 does not require Redis unless you want to use Action Cable.
Re:Rescuing Resque again (2015) by bdcravens on Friday January 15, 2016 @01:37PM (#10911142)
Interesting to note that the current stable branch has commits a couple of days old.
Re:Rescuing Resque again (2015) by steveklabnik on Friday January 15, 2016 @02:03PM (#10911331)
Yes, I've been doing some work, and so have some other contributors. Shopify is upstreaming some patches so they can get back on master.