Do these results have any real world value?
  
  Probably not. (...)

Things like that always remind me of this great quote by Tom Lane (source: http://en.wikipedia.org/wiki/Tom_Lane_%28computer_scientist%...):

  On idiotic benchmark comparisons "Try to carry 500 people
  from Los Angeles to Tokyo in an F-15.
  No? Try to win a dogfight in a 747. No? But they both
  fly, so it must be useful to compare them... especially
  on the basis of the most simplistic test case you can
  think of. For extra points, use *only one* test case.
  Perhaps this paper can be described as "comparing an F-15
  to a 747 on the basis of required runway length".

Absolutely. It's also true the other way - you don't have to use ActiveRecord with Rails, or the native ORM with Django, etc.

Of course, we already know that these benchmarks don't really have any parity, but they're interesting at a very high level as long as you understand why one might be faster.

PS: A more complex test would be nice, though - at a cursory glance, Django's ORM doesn't appear to be as slow as it used to be/is often attributed to be.

When building a company it's also good to choose something your team-members know and you can recruit for.

For that same reason, I think metrics on how much RAM they settle in at would also be interesting. They could disqualify some frameworks from RAM-constrained requirements right away (ex. if you're building some hardware with an embedded-but-sophisticated web ui).

First, the version of Go they're using (r59) is very old. The current stable release is r60, and most Go users have recently switched to tracking the weekly snapshots in preparation for Go 1 (mere weeks away).

Second, and most importantly, they demonstrate the web.go and mustache.go libraries. These third-party libraries were written shortly after Go was released (in 2009 or perhaps early 2010). Since then, Go's standard http and template packages have seen a lot of development and should now perform much better than web.go and mustache.go.

Third, there's a new "database/sql" package in the recent weeklies that provides a single interface to SQL databases. There are several drivers available, including sqlite, so it's relatively easy to implement the database part of the benchmarks in Go, too.

Given the task of benchmarking frameworks, I suppose the author thought it necessary to reach for a framework. That explains why he looked outside the standard library for these tools. Fortunately for Go programmers, the Go project regards http, templates, and databases as core functionality in its standard library.

http://weekly.golang.org/pkg/

If you want to see some impressive figures from a compiled language, look at Haskell's snap.

Go isn't just about speed. it is also a much cleaner and simpler language than Python. Go is designed to scale programs from tens to tens of millions of lines of code.

I jumped on github to look at some go code after you said this, but was quickly dissapointed. It seems like a mix between java and c

What I wonder more about is if the benchmark is setting the GOMAXPROCS env variable according to the number of CPUs on his system, otherwise he has 4 cores and 3 are idling around...

I remember a test that compared MySQL to SQL Server (Microsoft's SQL Server) a few years back.

MySQL had SQL Server beat hands down... For 1 concurrent user.

Once things scaled passed 10-20, SQL Server started winning.

At 30-50, MySQL could no longer respond and would crash, while SQL Server scaled to 90!

But everyone was wowed by the 1 user-load test! And countless posts were made over the years showing how MySQL is superior to the obviously flawed SQL Server.

Just curious, how many?

That's usually a worthwhile tradeoff (hardware is cheap) but one that a growing number of junior devs don't seem to be making consciously anymore. For them it's just normal to be spending more time per request in the controller code than in the database query. Whereas for many oldtimers it will never cease to feel a little funny...

A side-effect of this trend is that you can't reasonably run your beefy rails-frontend on a mediocre CPU anymore. When latency is dominated by ruby-code crunching on strings then you really want the fastest per-thread performance that money can buy.

Secondly, let me reiterate (since some people don't seem to be reading the website that's home to these benchmarks) that while I do think benchmarks like these can be interesting, I don't believe that they carry much real-world value. As the website states:

  "When it comes to code, the slightest adjustments have the potential to change
  things drastically. While I have tried to perform each test as fairly and
  accurately as possible, it would be foolish to consider these results as
  scientific in any way."

As to the "why isn't XYZ listed?" questions: This started as a pet project of mine out of pure curiosity, so I initially only included frameworks that I was familiar with. Thanks to GitHub, I will try to add more frameworks as the pull requests come in as long as I feel like they are popular enough to deserve a place next to Rails, Django, etc. Keep in mind that I have listed the Amazon EC2 AMI that was used, so feel free to run your own tests if for some reason I opt not to include your favorite framework in this list.

Finally, the last refresh of these tests was done in September 2011, which should explain why some of the version numbers are a bit outdated (I didn't know putting this stuff on GitHub would get so much attention or I would have waited). I hope to have another refresh done soon (Django 1.4, Pyramid 1.3, Rails 3.2, CakePHP 2.1, and a few new faces) but life/work is busy at the moment so please be patient and feel free to run your own tests in the meantime.

Of course there is. The "hello-world" microbenchmark tells you something about the core stack performance of the frameworks relative to one another. This tends to be surprisingly indicative of the relative real-world throughput per node that you may expect for a more complex application.

I.e. Rails (ruby) scores 4 times lower than Python in that micro-benchmark. This is pretty close to the difference that I've observed between real applications on the respective platforms. If anything you'll see the difference magnified in a complex application, but it's unlikely to turn around.

Whether that matters or not in the big picture is a different question (it usually doesn't).

I wouldn't say so. The bottleneck in a hello world benchmark might be a component of the framework, but in complex applications it's likely to be something else. Just look at the "Template Test with DB Query": There, Sqlite is the bottleneck, and the performance difference between Django and Rails fades into background completely.

Nope. Ruby/Python app-servers always end up CPU-bound, unless you're doing something very much out of the ordinary (such as blocking on an embedded SQLite...).

It might actually be an interesting exercise to gradually layer more features on top (such as the dispatcher chain and different templating/ORM layers) to discover inefficiencies in the various frameworks by comparing the relative performance impact.

E.g. if you involve the dispatcher and Django suddenly gets 20% slower but Rails only 10% slower then that might hint at an inefficient dispatcher-impl in Django.

I've submitted a pull request[1] to rectify this gross oversight. Even though this benchmark is not particularly scientific, it still provides some meaningful data.

[1] https://github.com/seedifferently/the-great-web-framework-sh...

Given that Symphony had such a huge buzz (aside from CodeIgniter), it is apparently the slowest..

Rather harder to automate that, though. :)

Noir has been really fun to work with. It's really fast, API is predictable and easy to learn. It seems to be inspired heavily by rails but without the constant dependence on "magic" to do simple things.

It's also nice to break away from the Active Record overhead and be able to find new optimal ways to model data with Redis/Mongo.

I'd still recommend rails to the newbie developer, but if you're experienced in developing web apps, I'd highly recommend Noir.

The Php frameworks are brought up and torn down on each request - they are not daemons.

The Php frameworks here, use a lot of file reads (many include files), compared with some of the slimmer micro-frameworks.

This may ruffle some feathers of people who love their particular framework and derive satisfaction from using it over others....

I think some other questions it's bringing up are even more interesting:

- If no problem is the same, does any framework matter, except the fact that you have one that decently keeps your organized?

- Is there really a one framework fits all approach? I think not. Some benefit from ORM, some might not.

- We all know premature optimization is not productive. Does it matter what really anyone uses?

- If we don't like trivializing a particular framework to these metrics, why do we trivialize any other framework in general?

- How do we build a better comparison?

It's pretty easy to get a naive routing implementation perform really well with a handful of routes. The tables can easily turn in the context of a larger application however.

[1]: https://news.ycombinator.com/item?id=3592769

https://github.com/nascheme/quixote-shootout

It looks like mod_wsgi is quite a dog.

https://github.com/carbonfive/hellod/blob/master/results.md https://github.com/carbonfive/hellod/blob/61bcf7495470350ea1...

The OP benchmark is missing netty and other frameworks meant to be high performance, for higher performance languages.

Tornado is definitely a web framework.

If there are cases where they may overlap, how would you want to develop a test?