All benchmark details are linked in the Github readme. https://docs.google.com/document/d/1aDsS0V-AybpvXEwblBlahGLp...

For Redis - The benchmark client and remote server (ie AWS ElastiCache) were in the same AWS availability zone (us-west 2a) to minimize network latency as much as possible.

FYI it is a little suspect to write your own testing framework and then claim a ridiculous performance gain over something like Redis. There are already widely accepted testing tools [1] you could have used.

It might make more sense to write a harness that allows your software to be used with standard Redis wire protocol so it can be properly benchmarked and compared against the dozens of existing solutions in the space.

Also it seems like you accidentally discovered the Latency Numbers Everyone Should Know [2]. Local operations are faster than network operations.

1. https://redis.io/docs/management/optimization/benchmarks/

2. https://static.googleusercontent.com/media/sre.google/en//st...

Exactly! Accessing data in RAM will be orders of magnitude faster than over the network (maybe this is a good sanity check of our benchmark numbers). The core principle behind IKV is that it allows you to access (large) data-sets without network calls.

Most DB tooling out there only works for a client-server model.

And implementing the Redis protocol would imply changing our architecture significantly and negatively affect performance (ex. a producer-consumer queue to serve requests, ser-deserialization costs).

> implementing the Redis protocol would imply changing our architecture significantly and negatively affect performance

Yes, you’d be doing a fair and honest benchmark if you want to compare yourself to Redis.

I just explained how implementing the Redis protocol will inherit inefficiencies of Redis in IKV - so I don't understand how the comparison will be fair or honest.

Hope this addresses your original question about why we wrote a custom benchmarking client.

https://github.com/inlinedio/ikv-java-client/blob/master/src...

It is quite simple and is available here. There is nothing malicious in there to make IKV appear faster. Although ff you do see a bug, I am happy to fix and republish results.

> Load Generation

> We use a single benchmarking client to drive traffic to - (1) embedded IKV instance on the same machine as the client (2) A multi-sharded Redis cluster (built using AWS ElastiCache for Redis) in the same data-center (AWS availability zone). Using multiple clients is complex since we’re testing an embedded database (other benchmarking tools which only test standalone DB services can drive more load using multiple clients). The maximum amount of load that can be created depends on the number of parallel threads in the client and the average response time of the underlying database.

This is a ridiculous comparison: "embedded IKV instance" vs a cloud service.