I want the test to require the work of creating an object or allocating the memory for the response and then use of a JSON serializer.

These tests are intended to be stand-ins for realistic application behaviors. They are not realistic applications, but they should make use of the same functionality that a real application would. I want the tests to establish realistic high-water marks for the types of operations they exercise.

Running a single query per request? The high-water mark you might expect is what you see as our single-query test. It would be very difficult to make a single-query operation in a real application more trivial than what we have.

If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back.

Bottom line: nothing beats benchmarking your own application. But the tests in this project are intended to give a first-pass filter of sorts, assuming performance is on your list of requirements.

> If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back.

A better way to avoid this would be to make the output depend on the input. Require that the reply include some JSON that contains portions of the request.

Likewise for databases, require that the reply contain contents of a previous request. If you want to, make the request contain an ID and make the reply return the data keyed under that ID.

If you want to require that the data is persisted durably, require that the benchmark continue to function even if all processes are killed and restarted in the middle of a run.

What really matters about a system are its inputs and outputs. If an innovative system comes along that works differently inside, it shouldn't be disqualified from a benchmark just for being innovative.

That wouldn't work sufficiently well; unless the benchmark is quite complex it would almost always be better to implement an in-memory custom datastore rather than use a traditional backend. Even if you require persistance, since the intentionally small dataset means you can get away quite well with raw memory dumps or even an mmap variant. You wouldn't use a full-fledged JSON serializer but a limited string concatenation.

In any case, the way I read it, the requirement doesn't mean you have to make anything particularly heavy, just that you need to use a representation that's general enough to be usable in other scenarios without rewriting the entire app.

Of course, a little bit of extra encouragement in the form of a benchmark that's slightly harder to game wouldn't be bad.

Custom datastores (custom at the framework level, not necessarily at the application level) have been very useful for me in the past for getting high performance.

For example, I wrote a little in-memory DB for use in a .NET framework to avoid having to deserialize an object graph per request. Instead, it navigated a byte array and plucked out just the data needed.

Another time, I wrote a compact in-memory representation of a trie, using bit-packing and a few other tricks, to get an order of magnitude reduction in memory usage, making it possible to cache a lot more of a data set.

Why go to the database and instantiate a classic object if you can avoid it?

> Why mandate how the benchmark is implemented?

Otherwise you won't be able to compare the results.

For example, the Benchmarks Game [1] specifies how the algorithms in those benchmarks should work. For some of the benchmarks, there are faster algorithms. However, it's not about comparing algorithms, it's about comparing language implementations.

[1] http://benchmarksgame.alioth.debian.org/

You're going to stream a Hello, World value? Why use JSON, at all, when we can just print the resulting JSON text! Benchmarks should be defined, that's all.

> Why use JSON, at all, when we can just print the resulting JSON text!

Indeed. But if you want to actually force the server to encode some JSON, then make the reply include portions of the request. Then you don't have to have artificial rules about how it is implemented to get interesting/meaningful results: https://news.ycombinator.com/item?id=6650499

No, you still need to, because you can otherwise do a trivial string concat in most cases, even though in reality that would be most unwise for security, robustness, and maintainability reasons.

Since you don't know the inputs ahead of time, the benchmark would be incorrect if it could not handle the corner cases like special characters that need to be escaped in JSON strings.

If you wanted you could also make the request specify the key at which a value must be written.

Be creative. :) But don't think that apps always have to be written the way they are now to be "legitimate."

For JSON serialization, I'd assume it's because they're testing object serialization.

Right. Related to this, I feel like a couple of frameworks are cheating on the plaintext test because they aren't actually testing string... serialization. They encode the "Hello, World!" string into bytes once, retain those bytes in memory, and send those bytes on each request just to avoid encoding the string again.

It's such a small, silly optimization. It doesn't bother me because it's not going to affect the overall results much, but I do feel it's against the spirit of the test. That kind of thing would be prevented if we made the test echo a request parameter, as was suggested elsewhere in the comments.

Clients of a system don't know or care if the server actually constructed an object and serialized it. They care that they get the reply they expect, and do not care how it was constructed.

Right, but then you're just testing how fast a framework can vomit out a small string and some headers.

I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.

See: https://news.ycombinator.com/item?id=6650499

I wholeheartedly agree. But like said:

> I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.