This is a vital point — everybody wants to shoot for lovely, elegant systems that use minimal resources. But resources are astoundingly cheap; we've got a bunch of servers with 128GB of RAM, and you can get these for around £100 a month each. That's less than two hours of developer time. If using Ruby in place of another language saves you that much time—or even just makes you a little happier—it's probably worth it.
"This is a function of how Rails does concurrency, which is by using multiple processes."
Multi-process vs multi-threaded is an application server architecture matter, not a Rails matter. Yes, your application must be thread safe, but this has been available in some form for Rails apps since a long time now [1]. The biggest problem in recent history has been Gems and their thread-safetyness (is that a word). The number of threaded Ruby application servers has been on the rise for a long while now, and even Passenger (traditionally the multi-process front-runner) is going threaded in version 4.0.
1: An interesting blurg post related to Config.threadsafe! (which was a huge source of confusion): http://tenderlovemaking.com/2012/06/18/removing-config-threa...
Edit: Our requirement was to process this queue as fast as possible and that means more workers. With process based concurrency that is very costly as you have explained.
Also, if the workers shared all the code, you could have used unicorn to fork the processes after the code loading was complete. The 400MB per process would then instantly come down to something ~10MB per process at which point rewriting would have been delayed for another year or so.
Unicorn forking benefit is overrated, we used it and we don't see much benefit for long running processes.
Sidekiq is good alternative but that means some rewrite(for our app anyway). Secondly Sidekiq looks mature today, I started working on some of these changes 2 years ago.
2 years ago, ruby was not COW friendly. So yeah, there was not much benefit to forking if you were using 1.9.3. Not sure how well does ruby 2.x fares in that respect
So, in any case you have to rewrite as much as code that I rewrote in Go and decoupled from main App. (its not lot of code, I mentioned in talk)
Btw how are you generating the PDF from HTML and are able split a single HTML into multiple PDFs?
And I'm not splitting pdf but splitting html generation work load, and then create individual pdfs from those html chunks. Then they will be joined together (using pdfunite). I found this much faster then joining html and generating large pdf.
The X is largely dependent on how many dependencies your app pulls in, on your garbage collection settings, and on how much state you carry around with you. I don't know what an off-the-shelf Rails app requires as a minimum, but as a comparable, BCC requires ~120 MB per process and AR requires approximately ~200 MB. Neither are particularly mammoth code bases.
Thus, if AR develops a business requirement for 100 requests per second on something which takes 200 ms per request, my server budget just increased by 4 GB of RAM. That is absolutely unavoidable in the Rails concurrency model. It doesn't matter how well you code the small thing at issue: you pay a memory tax based on desired maximum volume and app complexity, always.
(Neither here nor there, but AR has features quite similar to the notification feature that the app described in this article. We handle this by using queue workers -- which each require 200 MB, as described above -- and the decision that queues do not require immediate servicing. Instead, if thousands of calls are dumped into the system, we add them to a queue quickly and then consume the queue leisurely. Different apps are different apps, but I'd wager a guess that we do substantially more volume in terms of calls on ~400 MB of RAM.)
Also neither here nor there, but engineers are expensive and gigabytes are cheap. There exist many circumstances in which the architecture which requires 12 GB of RAM is a defensible engineering tradeoff versus the one which takes 5 MB. As Thomas told me once when I was asking him how to cut 100 MB off of Redis so that I could avoid upgrading a VPS by 1 GB: "How much does the extra gig cost? $10? Why are we even having this discussion?" And he was right.