The benchmark is done considering what an end user will use directly in their application. The performance gain comes from avoiding remote network calls and minimizing serde.

The landing page is under construction :)

This seems counterintuitive to an embedded store. Potentially 12 hour provisioning time is also wild.

In the larger picture I’m trying and failing to imagine the niche for the eventual product but that could be a lack of familiarity or imagination on my part. I’m guessing the OP is part of the team that’s working on this? If so, maybe you could elaborate on what specific problem this is solving? Additionally is there any possibility of self-hosting? Since writes do obviously involve network traffic, they’ll almost certainly be faster over a 6’ 10-Gbit SFP cable to the pool of NVMe drives sitting in the rack here.

Also, since the use case sounds like “datasets that can’t fit in RAM”, what’s the cold start latency like? Say I’ve pushed 10TB of data into IKV. How much does a given new node have to pull down into local storage before it can start reading from (potentially a shard of) the data?

The primary usecase for this is serving features for ML inference (since eventual consistency is ok and sacrificing write latency for reads is a fair tradeoff). Right now, this is done by using a traditional client-server DB at the moment (Redis/DynamoDB/etc) - or if you're a big tech company that cares about latency you can implement this on your own (https://doordash.engineering/2022/05/03/how-we-applied-clien...).

As far as self-hosting goes - yes writes will be definitely faster. IKV is fully open source so we're not opposed to it, just haven't figured out the details yet (since self hosting will mostly be useful to very large usecase)

At the core, we use in-memory hashmaps that reference memory-mapped files. So, when a dataset doesn't fit in RAM - it spills to disk automatically.

Cold start - the database is seeded with a "base image", that is built periodically by the backend. That's how a user can add new nodes to their cluster, and still avoid any RPCs.

That being said, if you don't have 10TB of disk, you have to partition IKV (and by extension your application). We support partitioning by allowing documents (the data) to declare partitioning keys. If one shard/partition cannot fit on disk - the store won't startup.

There are data pipelines behind the scenes to distribute writes to the embedded store which needs some provisioning time. And yes we are super early.

This a very bold statement.

For Redis - The benchmark client and remote server (ie AWS ElastiCache) were in the same AWS availability zone (us-west 2a) to minimize network latency as much as possible.

It might make more sense to write a harness that allows your software to be used with standard Redis wire protocol so it can be properly benchmarked and compared against the dozens of existing solutions in the space.

Also it seems like you accidentally discovered the Latency Numbers Everyone Should Know [2]. Local operations are faster than network operations.

1. https://redis.io/docs/management/optimization/benchmarks/

2. https://static.googleusercontent.com/media/sre.google/en//st...

Most DB tooling out there only works for a client-server model.

And implementing the Redis protocol would imply changing our architecture significantly and negatively affect performance (ex. a producer-consumer queue to serve requests, ser-deserialization costs).

Yes, you’d be doing a fair and honest benchmark if you want to compare yourself to Redis.

Hope this addresses your original question about why we wrote a custom benchmarking client.

https://github.com/inlinedio/ikv-java-client/blob/master/src...

It is quite simple and is available here. There is nothing malicious in there to make IKV appear faster. Although ff you do see a bug, I am happy to fix and republish results.

> We use a single benchmarking client to drive traffic to - (1) embedded IKV instance on the same machine as the client (2) A multi-sharded Redis cluster (built using AWS ElastiCache for Redis) in the same data-center (AWS availability zone). Using multiple clients is complex since we’re testing an embedded database (other benchmarking tools which only test standalone DB services can drive more load using multiple clients). The maximum amount of load that can be created depends on the number of parallel threads in the client and the average response time of the underlying database.

This is a ridiculous comparison: "embedded IKV instance" vs a cloud service.

Benchmarking a local program vs an AWS hosted service to claim you’re faster than Redis?

Reads are served locally but all writes go to some hosted cloud service?

Read path don't invoke any RPCs (even on startup or a cache miss). Writes need RPCs - since they have to be propagated to the (many) readers.

I like the idea that you can just use the software for free, or pay for cloud hosting.

But it seems like you need a cloud account even to use it locally.

The monetisation strategy seems to affect the product negatively.

Where is the FFI, I'd like to see how you do that and whether you go through C. I couldn't find it easily.

Duh, I don't know how I missed it https://github.com/inlinedio/ikv-store/tree/master/ikv/src/f...

Yup, it has JNI and C FFI