News for hackers. Stuff that matters.
Re:Web Framework Benchmarks Round 7 by integraton on Thursday October 31, 2013 @05:00PM (#6649844)
Summary: if you want performance, use Java, Scala, Go, Clojure, Lua, or C++.

Honestly, now with all the great Scala frameworks, Clojure, and the ability to run Rails, plus Cassandra, Storm, etc, I'm a little creeped out that I'm actually strongly considering building my current new project completely on the JVM.

Re:Web Framework Benchmarks Round 7 by recuter on Thursday October 31, 2013 @05:28PM (#6650083)
I'm a fan of Go and an even bigger fan of Lua, and I'm not sure I agree with your summary. Go and OpenResty can push 300,000 "Hello World" plaintext responses a second, and some Java stuff can do x2 that, so JVM frameworks are strictly better? And something called cpoll_cppsp trumps a sane language choice? Don't be cute. :)

It is perfectly fine to not squeeze every drop from your hardware and pick Go or Lua. In the same vain, Flask may be x20 slower at a meaningless benchmark but you'd note that the ones that involve actual work, you know, the kind your complex app will actually perform, the more involved it is the more the difference shrinks. All the way down to x4, in other words not an order of magnitutde.

The real takeaway summary should probably be: a well made framework like flask doesn't have so much overhead that the productivity gains it offers are not worth trading a bit of performance for.

A terribly made framework (I won't name names) will bite you in the ass. Choose carefully.

Re:Web Framework Benchmarks Round 7 by mangeletti on Thursday October 31, 2013 @05:37PM (#6650143)
I think the idea of a "meaningless benchmark" has become a dogma. Can you possibly know all the complexities that an app will contain when benchmarking a framework? No, but that doesn't make baseline benchmarks useless. Part of the reason for including the most basic (CRUD-ish) operations as separate benchmarks in their tests is so that you can make comparisons between basic "Hello, World" operations and more complex operations, which gives you a bit more insight into the overhead of each framework.

I'll take that any day, over guessing which framework is fastest.

Re:Web Framework Benchmarks Round 7 by recuter on Thursday October 31, 2013 @05:54PM (#6650268)
Agreed, I should have avoided that particular phrasing.
Re:Web Framework Benchmarks Round 7 by piranha on Thursday October 31, 2013 @05:37PM (#6650144)
Actually, it's other way around. In 'JSON encoding' tornado and flask-pypy are 4x slower than compojure, for example, but in 'Multiple queries' they are 14x slower.
Re:Web Framework Benchmarks Round 7 by integraton on Thursday October 31, 2013 @06:45PM (#6650560)
Just for the record, my comment didn't intend to imply that JVM frameworks are strictly better in all cases, and I don't believe that at all. In fact, I'd argue that for the majority of applications, choosing any modern framework purely for benchmark performance is very foolish since all of them perform more than adequately.

Rails, for example, despite always having been "slow," performs far more than adequately for the vast majority of web applications, scaling patterns are well established, and it's obviously hugely productive for many, many developers and companies.

Re:Web Framework Benchmarks Round 7 by mbesto on Friday November 01, 2013 @05:28AM (#6652667)
> A terribly made framework (I won't name names) will bite you in the ass. Choose carefully.

Facebook and Twitter seem to be doing alright. Care to elaborate?

Re:Web Framework Benchmarks Round 7 by mgl on Friday November 01, 2013 @05:39AM (#6652700)
Well, it really depends on the application type. We are a Java and Ruby on Rails software house (http://codedose.com), we also have experience working for investment banks, and for some cases Java with Spring or Play is the best way to go, especially if you think about apps with complex back-end.

We have recently released an online market for physical gold trading capable of handling 10k+ concurrent users with horizontally scalable architecture and complex trading engine/accounting logic, and it's all Java. It has to be fast and there is not enough static or almost-static content to make caching effective.

Re:Web Framework Benchmarks Round 7 by hbbio on Thursday October 31, 2013 @05:58PM (#6650290)
Too bad memory is not measured/displayed as well. For many cloud-hosted servers, you will end up paying for memory, rather than CPU.

And Java/Scala are disqualified pretty quickly.

Re:Web Framework Benchmarks Round 7 by rsynnott on Thursday October 31, 2013 @06:04PM (#6650320)
Java and Scala would have large base memory requirements, but not necessarily large per-concurrent-request memory requirements.
Re:Web Framework Benchmarks Round 7 by TylerE on Thursday October 31, 2013 @07:05PM (#6650693)
When ever I feel like that, a few hours with either the Java or Scala persistance frameworks rapidly cures me of it.
Re:Web Framework Benchmarks Round 7 by terhechte on Friday November 01, 2013 @03:47AM (#6652395)
This. In my last project I tried to use scalatra with slick, and while I really liked scalatra, slick made me go nuts. I had to jump over so many hoops that it was just a pain.

I'm currently working on a new project (still research phase) where I was pondering going with Spray and a lightweight Postgresql wrapper because I primarily need to read data from a database, do some transformations on it, and write it out as JSON as fast as possible. I had it working in Spray, but I had issues with server crashes and the speed wasn't as I'd have expected. I fiddled with it for one day, and then, out of frustration, decided to give openresty a try. I've never written much Lua in my life, but after only a couple of hours, I had it working, and it was far, far faster than the Spray implementation. I did some research there, and it seems that the database stuff took a whole lot longer in Scala/Spray than in Lua. Now, of course, I loose type safety, so there may be hidden issues in there, but since I'm really just doing simple data transformations, I think I'm fine with lua / openresty.

As a first verdict, I really, really like what I've seen of Openresty so far.

Re:Web Framework Benchmarks Round 7 by EtienneK on Friday November 01, 2013 @02:44AM (#6652195)
Honestly, JPA is not that bad.
Re:Web Framework Benchmarks Round 7 by flavor8 on Friday November 01, 2013 @06:22AM (#6652822)
mybatis.
Re:Web Framework Benchmarks Round 7 by thezilch on Thursday October 31, 2013 @05:55PM (#6650276)
Or PHP, as you're most likely hitting some kind of DB / cache and not just serving plaintext or cached JSON out of your backend... backs away slowly
Re:Web Framework Benchmarks Round 7 by jol on Friday November 01, 2013 @04:41AM (#6652507)
Actually I was quite surprised that multiple queries benchmark showed more than 1 entry for PHP in top 10. take-away: if you don't do much sophisticated data processing then PHP is ok choice. gets ready to run
Re:Web Framework Benchmarks Round 7 by Joeri on Friday November 01, 2013 @04:53AM (#6652548)
That's because php is a really thin wrapper around the underlying C libraries. The thing that makes it ugly is what makes it fast. Php is still interpreted and not jit'ed, even with the opcode cache used in this benchmark, so it loses out in anything that exercises the php code side of the equation, which is why the big php frameworks (symfony, cake and laravel) are 50 times slower than raw php. I wonder how facebook's hhvm engine would stack up, as it is a jit'ing php engine and supposedly an order of magnitude faster.
Re:Web Framework Benchmarks Round 7 by bhauer on Friday November 01, 2013 @09:02AM (#6653345)
That's my understanding of the situation as well.

As a related aside, we have an intention to eventually capture some additional statistics about the implementations such as source lines of code, total number of commits, and possibly lines of code of libraries (where available).

These are just additional data points to use as the reader sees fit, but they may provide some insights. For example: (a) developer efficiency, assuming you are comfortable using sloc as a proxy for developer efficiency, which is admittedly a hotly debated matter; (b) commits are a proxy for the level of attention the particular test implementation has received in our project, since a test with only 1 commit may be unrefined and in need of tuning and a test with dozens of commits may be considered highly volatile; (c) where applicable, the performance impact of additional code, perhaps even as granular as a calculated average per-line cost in rps. The last point might be particularly illuminating for languages such as PHP.

Re:Web Framework Benchmarks Round 7 by mgkimsal on Friday November 01, 2013 @04:18PM (#6656112)
http://blog.liip.ch/archive/2013/10/29/hhvm-and-symfony2.htm...

Doesn't look to be "an order of magnitude faster" - 200% faster in some cases - certainly nice numbers, but not a massive game changer for many (yet?).

Re:Web Framework Benchmarks Round 7 by ohashi on Thursday October 31, 2013 @11:28PM (#6651618)
php-raw performed quite well.
Re:Web Framework Benchmarks Round 7 by andypants on Friday November 01, 2013 @08:59AM (#6653336)
As soon as you throw a php framework on top though, it sinks to the bottom of the benchmarks.
Re:Web Framework Benchmarks Round 7 by fixxer on Thursday October 31, 2013 @10:39PM (#6651499)
I'd run.
Re:Web Framework Benchmarks Round 7 by stesch on Friday November 01, 2013 @11:48AM (#6654550)
If you ignore Erlang and run your database on a ramdisk.

Sorry, round 7 isn't helping much. Wait for round 8.

Re:Web Framework Benchmarks Round 7 by arocks on Thursday October 31, 2013 @08:34PM (#6651079)
or Haskell :)
[deleted]
Re:Web Framework Benchmarks Round 7 by cgh on Thursday October 31, 2013 @05:12PM (#6649956)
Here is the environment info:

"As of Round 7, three Intel Sandy Bridge Core i7-2600K workstations with 8 GB memory each (early 2011 vintage) for the i7 tests; database server equipped with Samsung 840 Pro SSD Two Amazon EC2 m1.large instances for the EC2 tests Switched gigabit Ethernet"

So 8GB workstations. Even assuming full memory usage, the JVM, Go and C++ frameworks destroyed all comers.

Re:Web Framework Benchmarks Round 7 by rallison on Thursday October 31, 2013 @05:28PM (#6650080)
Just for reference, the deleted comment used to say:

Raw performance is nothing without memory usage information.

Re:Web Framework Benchmarks Round 7 by TomasEkeli on Thursday October 31, 2013 @05:13PM (#6649960)
That said - memory is dirt cheap
Re:Web Framework Benchmarks Round 7 by mgkimsal on Thursday October 31, 2013 @06:09PM (#6650348)
There are still practical limits, and sometimes you have to work within those. I've got a project on a 16G server - the data center doesn't have anything bigger. Moving everything to a custom server or different data center to get to 24G or 32G is... a lot of work. Some time spent finding memory reductions is still worth it.

And what if we went to 32G or 64G, but still needed more? "RAM is cheap" doesn't scale. The hardware needed to host a 128G system (or multiple 16/32G systems) isn't cheap.

Re:Web Framework Benchmarks Round 7 by sciurus on Saturday November 02, 2013 @01:05PM (#6659820)
Either you're wrong about the cost or we have a different definition of "cheap". The sweet spot for RAM pricing right now seems to be 16GB DIMMs, which cost $150 to $250. I can go to dell.com and configure a R320 with a Xeon CPU and 96GB of RAM for $2,500. That's less than a high-end Macbook Pro! A R420 with two Xeon CPUs and 192GB of RAM costs $4,300.

I recently upgraded two Dell Poweredge 2950s from 16 to 32GB of RAM. The cost per server was $300. If I'd needed to, I could have upgraded them to 64GB. That's on a five year old server that sells refurbished for $800.

Re:Web Framework Benchmarks Round 7 by Thaxll on Thursday October 31, 2013 @05:31PM (#6650097)
Not really, on a desktop it is, on Amazon it's very expensive.
Re:Web Framework Benchmarks Round 7 by meepmorp on Thursday October 31, 2013 @06:32PM (#6650487)
So don't use EC2 with your java apps.