News for hackers. Stuff that matters.
Re:Web Framework Benchmarks Round 4 by zinxq on Thursday May 02, 2013 @02:14PM (#5646228)
There should be no surprise that interpreted, dynamic languages are utterly out-gunned as compared to compiled (JIT'd or otherwise) languages. It's inherent to the system - every little thing you do costs more.

Many people choose Ruby and figure, given that premature optimization is the root of all evil, they'll optimize later if needed.

That's like choosing between a farm tractor or a ferrari - and figuring if the tractor doesn't perform up to snuff, we'll add a spoiler (and given the 10x disparity between Java and Ruby in some of those graphs, if we throw out a 20mph top speed for a farm tractor, the ferrari analogy is actually rather spot on).

There are many good reasons to choose dynamic/interpreted languages - but always know you're giving up performance in exchange.

Re:Web Framework Benchmarks Round 4 by dragonwriter on Thursday May 02, 2013 @02:19PM (#5646251)
> Many people choose Ruby and figure, given that premature optimization is the root of all evil, they'll optimize later if needed.

True.

> That's like choosing between a farm tractor or a ferrari - and figuring if the tractor doesn't perform up to snuff, we'll add a spoiler

Its really not like that at all, because programming languages aren't like vehicles. Particularly, with Ruby, on typical method of optimization is finding which bits of code are bottlenecks, and then optimizing those bottlenecks, often by replacing them with C (or, if the Ruby runtime being used in JRuby, Java).

Which I guess is like having your tractor turn into a Ferrari for the parts of work that involve going long distances on a road without towing something, but I think that kind of points out how bad even using the tractor/Ferrari analogy is.

Re:Web Framework Benchmarks Round 4 by candybar on Thursday May 02, 2013 @02:29PM (#5646309)
I keep hearing about this, but do people really rewrite performance-critical parts of their web apps in C? Even if it happens to be part of some third-party library? And maintain a fork? What if that performance-critical part is dependent on other parts in a non-trivial way? It seems that an unanticipated replacement of some core functionality with a C library may involve a major rewrite and most Ruby teams may not have the expertise to do a good job maintaining a C code base any way.
Re:Web Framework Benchmarks Round 4 by bnferguson on Thursday May 02, 2013 @03:28PM (#5646621)
Yup! Many do. Github just wrote about it here (replacing the Rails default HTML escaper with a C one for a 30% increase): https://github.com/blog/1475-escape-velocity and I think the Judy Arrays they're using for code classification are in C.

At my job after benchmarking we've done things like break out computation heavy things into C/C++, and have been even eyeballing things like Go and the Lua/Nginx based OpenResty for small computationally heavy services.

In many cases this means rewriting what used to be a 3rd party library. The big question is usually around cost in time and if we want to have to maintain that knowledge for the long term. Most of the time it's cheaper to toss more servers at it - but for certain things - namely cases where latency is very important no amount of scaling out is going to make it faster.

Re:Web Framework Benchmarks Round 4 by sbov on Thursday May 02, 2013 @02:49PM (#5646401)
I've never heard of it for webapps. They usually just buy more instances/servers.

It would be interesting to see the profile of some these benchmarks for the various frameworks to see where the bottleneck is.

Re:Web Framework Benchmarks Round 4 by dragonwriter on Thursday May 02, 2013 @02:53PM (#5646440)
> I keep hearing about this, but do people really rewrite performance-critical parts of their web apps in C?

Certainly they do it for Ruby apps in general. I don't think its all that common for it to be a high-value proposition for web apps.

> Even if it happens to be part of some third-party library? And maintain a fork?

If its an open-source third-party library that tends to get used in a way that is performance-critical, upstream will probably accept moving bottlenecks to (portable) C and maintaining the API, so its unlikely that you'll need to take up responsibility for a fork.

> What if that performance-critical part is dependent on other parts in a non-trivial way?

If the call pattern is such that they are not part of the performance critical part themsellves, then the performance critical part calls them through the regular conventions for calling Ruby from C.

If the call patter is such that they are part of the performance critical piece, well, I think the answer is obvious.

> It seems that an unanticipated replacement of some core functionality with a C library may involve a major rewrite

It might, but in the meantime you've got working code.

> and most Ruby teams may not have the expertise to do a good job maintaining a C code base any way.

If the team determines it needs expertise in a particular area that it doesn't currently have, then it should either develop that expertise or bring in people that have it. That's true whether its particular domain expertise (e.g., building messaging systems) or particular technology expertise (e.g., C). That's part of the normal development of a team.

Re:Web Framework Benchmarks Round 4 by candybar on Thursday May 02, 2013 @03:16PM (#5646545)
I guess my point is that it seems disingenuous to point out "you can write that bit in C" as a way to mitigate performance problems with Ruby, when in practice it's so costly compared to available alternatives (throw more hardware, write manually optimized Ruby, switch to a faster language/runtime) that almost no one does it. How much of Rails is written in C? It's like proposing compiler extensions/patches as a way of dealing with performance problems. And if you have a complex application that utilizes many of Ruby's idioms to deal with the complexity, it's extremely unlikely that you can simply replace parts of it with C libraries without reorganizing in such a way to increase complexity.
Re:Web Framework Benchmarks Round 4 by dragonwriter on Thursday May 02, 2013 @06:54PM (#5647716)
> I guess my point is that it seems disingenuous to point out "you can write that bit in C" as a way to mitigate performance problems with Ruby, when in practice it's so costly compared to available alternatives

I don't think its costly compared to available alternatives; I think its generally an efficient alternative for the type of bottleneck that is actually related to implementation language efficieny. I think, for most typical web apps, the bottlenecks are only rarely of that type, so that's generally not where the effort is going to be spent, but for the ones that do have bottlenecks of that type, its quite appropriate a way of solving it.

> throw more hardware, write manually optimized Ruby, switch to a faster language/runtime

If writing manually optimized Ruby is an effective and cheaper solution, you aren't experiencing the class of bottlenecks that are related to implementation language efficiency. Switching languages or runtimes for a component is a proper subset of the work of switching languages or runtimes for a project, so the latter isn't going to be less costly than the former (it may, if language-related bottlenecks are pervasive, or if you have non-performance interests in the alternative language, have a bigger net payoff and be more cost effective, but it won't be less costly, and its inherently riskier to do all at once, since component-wise transition gives you a faster cycle time in terms of realizing value even if you end up doing a full replacement in the end.)

> And if you have a complex application that utilizes many of Ruby's idioms to deal with the complexity, it's extremely unlikely that you can simply replace parts of it with C libraries without reorganizing in such a way to increase complexity.

I disagree. Anything you can do in Ruby you can do in API-equivalent C that can still call out to the exact same Ruby code for the functions that aren't being moved into C, so there is no reason at all for the kind of reorganization you suggest, particularly if you are building with loosely-coupled components in the first place.

If you are building a complex app and its all tightly coupled, you've got a big maintainability nightmare no matter what language you're using, and that has nothing to do with Ruby.

Re:Web Framework Benchmarks Round 4 by candybar on Friday May 03, 2013 @10:19AM (#5650160)
I completely disagree that Ruby's slowness is rarely the bottleneck. In this benchmark, we have reasonably simple requests on decent hardware in the realm of 1 second latency, where faster frameworks are 10+ times faster. We have Sinatra being far slower than similar frameworks like Scalatra, Unfiltered.

Yes, you can write Ruby in C, but it would be almost as slow as writing Ruby in Ruby. I don't really see the point of saying, you can do anything you can do in Ruby in C, it would be much more verbose and about as slow. The point is that true optimization may force you to do things that you can only do in C and there's no guarantee that this optimized version can be easily utilized from the rest of your Ruby code. This has nothing to do with tight-coupling - it's simply taking advantage of the language's abstraction facilities.

And no, having to write hand-tuned Ruby, as opposed to idiomatic Ruby, to get performance that can be had by writing, say, idiomatic Scala or Haskell is an indictment of slow implementations and prevents you from taking full advantage of the expressiveness provided by the language.

And that's before you get into things like your team may have to get bigger because you need a C/Ruby-extension expert, half the team not being able to understand a critical part of the code base (very few Ruby developers are reasonably competent in C), etc.

Again, the whole point is that Ruby's performance problems pose a real pain point. Yes, you can rewrite parts of it in C, yes you can mitigate by using gems written in C, yes, you can spend more time optimizing, yes you can throw more hardware. But all of those are costly and it's disingenuous to pretend that a problem doesn't exist simply because a workaround does.

Re:Web Framework Benchmarks Round 4 by steveklabnik on Thursday May 02, 2013 @11:56PM (#5648728)
> How much of Rails is written in C?

None, on purpose. We want maximum portability, and so the Rails defaults are Ruby-only on purpose. Of course, it's easy to add gems that replace things that are written in C or Java, depending on what makes the most sense for your platform.

Re:Web Framework Benchmarks Round 4 by candybar on Friday May 03, 2013 @10:05AM (#5650108)
Would it make sense to submit a rails-optimized pull request for this benchmark that replaces some key performance bottlenecks with appropriate C gems? I'd be curious to see how fast rails can go out of the box without doing your own hand optimization.
Re:Web Framework Benchmarks Round 4 by steveklabnik on Friday May 03, 2013 @01:02PM (#5651139)
It's quite possible, I'm not sure about what specifically is slow in these benchmarks, because I haven't done any profiling.
Re:Web Framework Benchmarks Round 4 by pekk on Thursday May 02, 2013 @06:19PM (#5647548)
If you cannot write one small part of the app in C due to the difficulty or time consumed, then how much better is it for you to write everything in Java from the beginning? Java does not really substitute for Ruby in the same niche.
Re:Web Framework Benchmarks Round 4 by candybar on Friday May 03, 2013 @10:07AM (#5650110)
There are a bunch of languages that are about as expressive that are much faster, like Scala, Haskell or even Javascript.
Re:Web Framework Benchmarks Round 4 by binarysoul on Thursday May 02, 2013 @02:43PM (#5646382)
In many cases it is as easy as relying on a gem instead of doing it yourself. Where the gem contains a c extension. Say resizing images with chunkypng vs mini_magick
Re:Web Framework Benchmarks Round 4 by apalmer on Thursday May 02, 2013 @04:16PM (#5646940)
Seems a big part of this is is a lot of the proponents thought the comparison was really a Ford vs a Corvette, I dont think a lot of people were really internalizing exactly the ramifications of the order(s) of magnitude difference in performance. Which is why this kinda benchmark is pretty helpful.
Re:Web Framework Benchmarks Round 4 by justincormack on Thursday May 02, 2013 @04:56PM (#5647173)
Openresty is doing pretty well with Lua and despite the fact it uses LuaJIT it is actually interpreted and the JIT compiler is not used (that will change eventually). Ruby is just slow, as is PHP.
Re:Web Framework Benchmarks Round 4 by continuations on Friday May 03, 2013 @03:02PM (#5651953)
> despite the fact it uses LuaJIT it is actually interpreted and the JIT compiler is not used

Why not - is that a limitation of OpenResty or of LuaJIT?

How would you turn on JIT compiler in OpenResty?

Re:Web Framework Benchmarks Round 4 by justincormack on Saturday May 04, 2013 @08:08AM (#5654800)
The traditional Lua API is interpreted only so every call interrupts a trace. You have to use the ffi API to call C code instead. Plus some string operations are interpreted only but that is being fixed. It is all being worked on but there is a fair amount to do...
Re:Web Framework Benchmarks Round 4 by meric on Friday May 03, 2013 @12:25AM (#5648800)
Can't wait to see Openresty benchmark with LuaJIT.
Re:Web Framework Benchmarks Round 4 by jakejake on Thursday May 02, 2013 @03:26PM (#5646603)
I think the tractor/ferrari analogy does't really hold up because a tractor to me implies that it would be slower, but more powerful. The ferrari sacrifices power for speed. I'd say these platforms are more like comparing a ferrari with a go-kart. Or comparing a ferrari with another ferrari that has 10,000 lbs of bricks in the trunk. (Actually, that probably doesn't hold up either because ferraris probably don't have trunks!)

But to further add to the analogy, the tractor, the ferrari and the go-kart may all perform about the same if you're only traveling 1 inch.

Love me some analogies!

Re:Web Framework Benchmarks Round 4 by cmircea on Thursday May 02, 2013 @03:57PM (#5646808)
To add my 2 cents, static languages have started adding dynamic features. One example is C#.
Re:Web Framework Benchmarks Round 4 by camus on Thursday May 02, 2013 @02:16PM (#5646241)
People chose language X over language Y not for performance reasons. Cost , ease of use and deployment , librairies , available programmers ,etc ... Things are more complicated than just a benchmarks. Furthermore NodeJS and raw PHP are doing quite well in the benchmarks.
Re:Web Framework Benchmarks Round 4 by zinxq on Thursday May 02, 2013 @02:23PM (#5646274)
I did say exactly that - there's plenty of good reasons for choosing them - but understanding its a tradeoff is a good idea.

In other words, although interesting (and exceedingly well done) these benchmarks should have "surprised" no one. Not even the disparity between languages.