Redis vs. KeyDB vs. Dragonfly vs. Skytable
- Redis: https://github.com/redis/redis
- KeyDB: https://github.com/snapchat/keydb
- Dragonfly: https://github.com/dragonflydb/dragonfly
- Skytable: https://github.com/skytable/skytable
And, these are my benchmark results (30 threads, 1 client per thread with GET/SET. See below for setup):
1. Redis: 112,100 / 99,892
2. KeyDB: 288,931 / 282,997
3. Dragonfly: 408,322 / 392,446
4. Skytable: 619,992 / 676,091
# Notes
1. Redis: I'll start with Redis which I'd like to call the "original" key/value store (after memcached) because it is the oldest and most widely used of all. Being a long-time follower of Redis, I do know it's single-threaded (and uses io-threads since 6.0) and hence it achieves lesser throughput than the other stores listed above which are multi-threaded, at least to some extent. The best parts about Redis: it's the most feature complete of all the systems here, and is the oldest. (not saying old necessarily means better).
2. KeyDB: The second is KeyDB. IIRC, I saw it in a blog post which said that it is a "multithreaded fork of Redis that is 5X faster"[1]. I really liked the idea because I was previously running several instances of Redis on the same node and proxying them like a "single-node cluster." Why? To increase CPU utilization. A single KeyDB instance could replace the unwanted proxying funkiness, so I ditched Redis for KeyDB. Has been a fine experience so far, except for some occasional crashes.
3. Dragonfly: Just found it on HN and seems to be relatively new. Dragonfly claims that it is 25X faster (that I couldn't reproduce) than Redis[2] and has the slogan "Probably, the fastest in-memory store in the universe!". Doesn't support all the Redis commands yet, but I find it interesting mainly because of performance. Also, it's good to know why it is faster because it clearly outlines the underlying architecture[2]. The other three stores don't say much about it. Will be following the project.
4. Skytable: Found it while looking for projects written in Rust. Claims to be "insanely fast." Skytable's "experimental benchmarks" claim that it is something around 10X faster than Redis and some 2X-3X faster than KeyDB[3]. I hadn't heard of Skytable and it doesn't seem to be as widely used (unless I'm missing something?). I find it interesting because of the planned features[4] and performance. Only Skytable natively runs on Windows out of the four. Will be following the project.
5. My thoughts: Redis needs no introduction and is arguably super stable for use on production systems (using widely in our systems). KeyDB seems to be "stable enough" and it seems to be good for prod since Snapchat uses it already[5] (and so do we!). I found no Dragonfly v/s Skytable benchmarks. The best part about Redis, KeyDB and Skytable is that they don't make any "crazy assumptions" about the system they run on. What do I mean?
Dragonfly expects you to have the latest hardware[6] and the latest kernel[7] on all your servers. I find this outrageous because not all servers run 5.10 and a majority of them are still using long-running branches on 4.x. Also, I don't expect them to have the latest processors, either. I'd argue if the other three stores started assuming the latest features, they'd be far faster than what they are today. Finally, both Dragonfly and Skytable are still early in their development so it may not be fair to compare their features against Redis and KeyDB who have been around for far longer. Also, all projects other than Skytable have companies behind them (unless I'm missing something).
Edit #1: I have run the benchmarks myself for each store. In the benchmark with Redis, KeyDB and Skytable (redis-benchmark, memtier and sky-bench): Redis and KeyDB benchmarks seem to be very consistent, Skytable is a little inconsistent at times. However, in the benchmark with Redis, KeyDB and Dragonfly (with memtier), I was NOT able to reproduce the 25X speed that Dragonfly claims. I ran all tests on two m5.4xlarge servers (one with the k/v store and one with the benchmark tool).
Edit #2: Added benchmark results
What are your thoughts? Have you tried benchmarking any of them locally or in the cloud?
References:
[1]: https://docs.keydb.dev/blog/2019/10/07/blog-post
[2]: https://github.com/dragonflydb/dragonfly
[3]: https://github.com/ohsayan/sky-benches
[4]: https://github.com/skytable/skytable/issues/203
[5]: https://docs.keydb.dev/news/2022/05/12/keydb-joins-snap
The Dragonfly benchmark which showed 25x faster was run on c6gn.16xlarge which has 64 vCPU. The m5.4xlarge you used "only" has 16 vCPU. Since Dragonfly is achieving its performance by using multiple threads it's not weird that you weren't able to replicate its performance gain.
In my opinion, when it comes to these types of multi-threaded benchmarks it's much better to separate the "baseline, one-process performance" from "how it scales with number of processes". E.g. if you first pin Dragonfly to only run on a single core you can find the baseline performance compared to Redis, and then you can run different benchmarks of Dragonfly with increasing number of cores to see how it scales. This should give you much better understanding of how the system behaves. (The release of Dragonfly actually prompted me to write a blog post on this: https://t.holmium.no/dia/apples-vs-oranges/)
I don’t understand why this is “outrageous”? There’s performance features and better hardware utilisation in newer kernels? If you want the performance offered by the applications built on top of these, then arguably you should be prepared to run the hardware and software required to achieve this.
I don't see anything wrong with that. Either your kv store is performing OK or can be scaled using the usual methods and it's not holding you back... or you actually critically depend on the fastest kv performance you can get. In the second case, you'll be likely happy to dedicate some time to standing up a couple of machines with updated kernels.
> I find this outrageous because not all servers run 5.10 and a majority of them are still using long-running branches on 4.x
I don't think it's outrageous to ask people to upgrade 5.10. This was released over 1.5 year ago and there's no reason why your corporate overload shouldn't consider migrating to this, or at least have someone working on qualifying their workloads.
the command I used to load was: "memtier_benchmark --ratio 1:0 -t 28 -c 20 -n 100000 --hide-histogram --distinct-client-seed -d 256"
I did the recording to clear out any doubts whether my benchmark results are real.
Doing a fresh rewrite would have enabled a nicer threading model, but it would make it harder to maintain compatibility. Plus at the time it was more to suggest to Redis multithreading was both desirable and possible which were disputed back then.
Quick ask: I don’t see “some” of the other offering out there like MemCached… what was the criteria used to select these? I don’t see any source of how the test where run, specs of the systems, how the DB where set up, etc. Would be very valuable to have in order to attempt to re-validate these test on our own platform. I also came back and saw some of your updates to the outcomes, While the Get/SET numbers are provided, I don’t find a reference to time, resource utilization, or time which would IMO make the data more valuable in understating how the numbers where achieved. * I might be a factor of the benchmarks themselves, but would add value IMO to your fantastic post.
The key problem is building a firewall so that such people can focus. The *organizational* problems there are hard. If you have a firewall around five people, you can't tell if:
- They're sitting at home drinking beer and watching cat videos
- Are developing the world's fastest database
Which leads to meetings, presentations, funding proposals, and so on. Which, in turn, leads to the need for large teams.
I just want to cache some objects and avoid hammer the DB on every request :´(
https://aws.amazon.com/memorydb/
I feel like I'm missing something here. Which benchmarks did you run, and what were the results?
[1]: https://github.com/dragonflydb/dragonfly#benchmarks [2]: https://github.com/ohsayan/sky-benches
Are you looking for throughput or speed?
Also, what are your needs?
If I just wanted pure kv store i'd still go memcache. local install for stuff that can expire often (html template caches) and reach out to a network store for user sessions and the like.
all my kv caches are designed to be rebuilt and not care. if you need ACID kv go with a proper db.
My old friend Memcached just came by and want to say Hello. He is still alive and kicking. Netflix has been treating him well.
https://memc.rs/intro
I have been wondering for a long time if we will ever see Memcached 2.0 and MySQL 9.0
https://github.com/memcached/memcached/wiki/ProgrammingFAQ#w...
That's often a good choice!
A KVS can be arbitrarily horizontally scaleable. That might not sound necessary for a small project, but even there:
1. You can deal with big data (e.g. keystroke-level event data, etc.)
2. You don't have to worry about scaleability. It's one less thing to worry about.
3. You're set up for success if you take the unicorn route.
That's not memcached. I like memcached, but it's not the same as a KVS.
Forgot my dear buddy. Added it back.