Thoughts of incremental web app functionality to test:

1. Exercising a randomized mix of reading and writing. I think you already said you were planning a CRUD test. Consider a tunable ratio here, something like 10000 R to 100 U to 10 C to 1 D.

2. Exercising synchronous web service (JSONP) calls in two modes: (a) to some web service that is consistently fast and low latency, say, the initial JSON example from this test suite running in servlet mode, and (b) to a web service written in the same framework as the one being tested, again using the initial JSON example.

(The idea here is that many frameworks fall on their faces when confronted with latency. This is why synthetic tests are usually so poorly predictive of real world behavior -- people forget that latency causes backlogs and backlogs cause all parts of the stack to misbehave in interesting ways.)

3. Test async ability if the framework has it, with a system call (sleep?) that takes a randomized 0 - 60 seconds to return. Would help understand when a framework is likely to blow up calling out to a credit card processor, doing server side image processing, etc.

4. Exercising authentication (standardize on bcrypt, but only create passwords on 1 in 10K requests), authorization, and session state, if offered.

5. Exercising any built-in support for caching, where 1 in rand(X) requests invalidates the DB query cache, 1 in rand(X) requests invalidates the WS call cache, 1 in rand(X) requests invalidates the long term async system call cache, and 1 in rand(Y) requests blows away the whole cache.

For the enterprise legacy integrators, it would also be interesting to test XML as well (in particular, SOAP), anywhere we're testing JSON.

This is great input, Terretta. Exactly the kind of thinking I wanted to tap into.

Some quick thoughts in response:

For #1, the conceptual test included reading and writing in a 1:1 R:W ratio (well, to be more accurate 1:1 R:U). I like the idea of extending this a little bit to include C and D. For the sake of benchmark run-time, I'm looking to restrain the growth of permutations. But on the other hand, I like your idea of a tunable ratio. Something to think about!

I like #2 and #3 as well. I'll think about those some more too.

I really like the idea of incorporating some bcrypt and session state (#4).

We have a few caching tests in mind, but like elsewhere, we'll start out simple and then add complexity.

Thanks for the great input! This planning is to have some good long-term ideas in mind.

I have got to do a fork/pull before then in my infinite spare time but a quick question, why only ORM? Excuse my ignorance but there are almost no ORM that I've used which are not materially slower than straight SQL or json, the exception being g Salat on MongoDB. I wonder what not having an ORM would do to some of the poor performance experienced.

We expect that most modern webapps are (still) developed with an ORM, but we're not ORM hard-liners. A few of the tests are run without an ORM and are identified with the suffix "raw."