Right. Related to this, I feel like a couple of frameworks are cheating on the plaintext test because they aren't actually testing string... serialization. They encode the "Hello, World!" string into bytes once, retain those bytes in memory, and send those bytes on each request just to avoid encoding the string again.

It's such a small, silly optimization. It doesn't bother me because it's not going to affect the overall results much, but I do feel it's against the spirit of the test. That kind of thing would be prevented if we made the test echo a request parameter, as was suggested elsewhere in the comments.

Clients of a system don't know or care if the server actually constructed an object and serialized it. They care that they get the reply they expect, and do not care how it was constructed.

Right, but then you're just testing how fast a framework can vomit out a small string and some headers.

I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.

See: https://news.ycombinator.com/item?id=6650499

I wholeheartedly agree. But like said:

> I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.