romange
romange on Hacker News
Kudos on the project launch! I can share my experience from when I was evaluating multi-threaded architectures for Dragonfly back in 2022.
At the time, I had a choice: go down the KeyDB path and suffer from lock contention in real-world use cases, or go with a shared-nothing approach. The latter paid a somewhat heavy price for inter-thread communication, but when combined with a carefully chosen transactional framework could truly become a seamless Redis drop-in replacement. I chose the shared-nothing model.
"Dragonfly removes locks and pays the cost of message passing; KiviDB keeps locks but makes them so fine-grained that they rarely collide and never need to move a request between threads."
Reading this sentence, I immediately saw where it fits into the broader taxonomy of in-memory store architectures. KeyDB, Garnet, and KiviDB share the same foundation: employing fine-grained locks. The choice of programming language (Rust, C#, C++) is less important as it's the underlying architecture that dictates the trade-offs.
With lock-based approaches, features like MULTI/EXEC, Lua scripts, and multi-key operations introduce severe contention that becomes hard to manage. Furthermore, background tasks such as snapshotting, full replication sync, and memory defragmentation are more difficult to implement efficiently when relying on low-level locks.
On the other hand, I would definitely opt for a shared-everything architecture like KiviDB's if I were designing Memcached today - a minimalist key-value store without complex multi-key commands, replication, or snapshotting requirements.
In any case, best of luck with the project!
Valkey/Redis support offloading of io processing to special I/O threads.
Their goal is to unload the "main" thread from performing i/o related tasks like socket reading and parsing, so it could only spend its precious time on datastore operations. This creates an asymmetrical architecture with I/O threads scaling to any number of CPUs, but the main thread is the only one that touches the hashtable and its entries. It helps a lot in cases where datastore operations are relatively lightweight, like SET/GET with short string values, but its impact will be insignificant for CPU heavy operations like lua EVALs, sorted sets, lists, MGET/MSET etc.
Dragonfly supports both epoll and iouring, and polling engine choice is quite orthogonal to its shared nothing architecture. I do not think that Valkey or Redis will become fully multi-threaded any time soon - as such change will require building something like Dragonfly (or use locks that historically were a big NO for Redis).
(Author of Dragonfly here)
I apologize. I must say that my tone, as you rightly wrote, was inappropriate.
Indeed, I am the lead developer for Dragonfly. As such, I am deeply concerned with the performance aspects of our product. Dragonfly claims to be a drop-in, better performant replacement for Redis and Memcached. Every test & benchmark we've run on multiple cpus reinforced that. I've never faked or tweaked any of these benchmarks. That is, of course, not an excuse, and is why I opened by apologizing. I'd like to take this opportunity, if I can kindly ask so, to learn what made your results differ so much from ours. I'll personally try to reproduce what you described. If you could also reach out to me, I'd be happy to learn more about the environments in which you've conducted the aforementioned tests.
I understand what you are saying. How would you suggest to present it then?
Dragonfly is not faster than Redis when running on a single cpu. It can not be, just because it has the overhead of the internal virtualization layer that composes all the operations over multiple shards (in general case). But Dragonfly can scale vertically with low latency and high throughput unlike other attempts of making multi-threaded Redis that used spinlocks or mutexes. So how do we demonstrate the added value?
Absolutely. And I am sorry I reacted this way.
1. We removed the FAQ page
2. You are right, I need to shut the fuck up and let self-righteous hn crowd with torches do what they do best - find a weakness, and push it until they get bored and switch to beat another builder. You asked me about my perspectives and priorities? These are my priorities:
https://github.com/dragonflydb/dragonfly/graphs/contributors
3. The first thing I did was on this thread was to apologize for having FAQ with such content.
You are an exception, then :)
But I still stand by the claim that fragmenting your stateful workload (i.e. Redis) into bunch of processes instead of having a single endpoint per instance is an acceptable approach in 2023. When your processes are excessively tiny, their load variability overshadows their average load. This imbalance results in unpredictable pauses, latencies, and Out of Memory (OOM) issues. This primarily occurs due to the absence of resource pooling under a single process. While it's challenging to exhibit this issue via synthetic benchmarks, it's certainly present.
Because we focus on building the most
awesome datastore and you should decide whether you prefer having your store designed by good marketeers or good engineers.
hmm and ships are designed to sail, yet you use planes to cross atlantic.
Nokia was designed as strongest and most affordable phone, yet you use Iphone that costs 1000$. it's not about how it was designed but whether it addresses your current needs. Developers do not want to manage a cluster of single cpu processes. Not on their laptops and not in the production. And it's not just about management complexity. See this
https://github.com/dragonflydb/dragonfly/issues/1229 and it's just one example. Single cpu - is just not enough for today use-cases.
We do not fight off capitalism in any way. Rather, I view capitalism as a powerful engine driving innovation. My experience as an engineer at both Google, a massively profitable entity that contributes significantly to the open source community, and AWS, which effectively utilizes open source software to bolster its own success, has shaped this perspective. I harbor no resentment towards AWS; instead, I see both approaches as legitimate strategies within the scope of a capitalist system. I firmly believe that companies in the open source space need to rise to the challenge of competition and devise effective strategies to thrive in the marketplace.
We are reading this and we will fix it shortly.
You are absolutely right, and I apologize for having this experience.
SEO is part of the game, but it should not be in FAQ section.
I think using word "misleading" is also "misleading".
Dragonfly hides complexity. Docker hid complexity of managing cgroups and deploying applications. S3 hid complexity of writing into separate disks. But you do not call S3 or minio misleading because they store stuff similarly to how disk stores files. Dragonfly hides complexity of managing bunch of processes on the same instance and the outcome of this is a cheaper production stack. What do you think has higher effective memory capacity on c6gn.16xlarge: a single process using all the memory or 40 processes which you need to provision independently?
KeyDB implements multiple threading with spin-locks that protect a global shared data structure.
Dragonfly is built upon shared-nothing architecture where each thread manages its own slice of data, hence no need for classical locks and no contention under high load. It still provides atomicity guarantees but allows multiple transactions to progress independently as long as they do not need exlusive access to the same keys. So basically different approaches to the same promise - scale. Also different trade-offs. Shared-nothing approach has less contention and more flexible transaction framework but inhibits a slightly higher 50%th percentile latency (order of 30usec).
It compares a process with a listening port with another process with a listening port. To give another example - nobody compares minio with bunch of disks to which you can write separately, and probably more efficiently.
Not yet but it supports JSON API
It does not have to be instead of our startup, really.
I made an honest mistake. While I saw DragonflyBSD, I didn't realize its prominence within the BSD community. I assumed that since our project only focuses on Linux, having a similar name in a different "namespace" wasn't a significant issue.
DragonflyDB cofounder here. I am not shy about our choice of license.
Like with software design, everything is about trade-offs. Folks here voiced reasons why we chose BSL. I am sure you perfectly aware about all this.
I do not know personally you but I noticed that you posted the link to the announcement. I am guessing you are passionate about the technology and innovation. Dragonfly is much more than the licensing choice we made. I wish HN discussions here were about how fibers work in Dragonfly and how SSD tiering is gonna be implemented and how we provide atomicity for lua scripts while running many of them in parallel etc.
Btw, Dragonfly relies on an io-engine called helio (roughly equivalent to tokio) that has been developed by me and open sourced under Apache 2.0.
Dragonfly is better with
1. Memory utilization (read the announcement, they mention it)
2. Deployment/orchestration - your initial threshold that forces you to scale horizontally just went up by order of magnitude. In fact for many use-cases you will never need to go horizontally with Dragonfly.
3. Dragonfly also provides a better experience when working with the system. Just a week ago one of the community contributors submitted a PR that introduces automatic recognition of the hot keys:
https://github.com/dragonflydb/dragonfly/pull/951 (this feature is not ready for production use yet but we will get there).
It also has a built-in open-metrics support, built in cpu-profiler support, fully asynchronous I/O that allows answering INFO commands even under load etc.
BSL license is more company-friendly than AGPL.
Yes, you can - in fact it supports out of the box existing frameworks and clients like redisson, sidekiq and many others
I did not claim that DrashCache is "the absolutely best" algorithm.
But some caching algorithms could be better than others on real-world data.
Would you want me to run something like https://github.com/twitter/cache-trace on both Redis/(LRU,LFU) and Dragonfly and see which one has higher hit-rate for the same memory requirements? Would this be a good test to decide if Dragonfly improves on Redis caching quality?
Backward compatibility is good? I disagree with you regarding this specific case. Redis philosophy is to add more and more settings, requiring from an end-user to understand internals and low-level trade-offs. And I am sorry, but in the case of caching use-case, a user wants it to "just work" and it's a reasonable expectation. I've run Redis myself in a startup I worked at - I tried LRU, LFU and it did not work very well due to mixed traffic patterns. And I even did not know if it did not work well due to LFU/LRU heuristic or due to the fact that Redis does random sampling and evicts a small number of items it tests. Eventually, I gave up and moved on. I am sure most users do the same or try increasing its cache capacity to ridiculous sizes.
So yeah, I do not completely ignore Redis LFU, I just did not find useful to mention it
because from my experience, Redis LFU is not being widely used, and does not have significant advantage over Redis LRU (if at all) and I wanted to maintain the focus and keep my post succinct and clear.
I also have not provided any benchmarks in the post, not because I have something to hide, just because I do not have an easy setup to benchmark cache efficiency vs Redis and I do not have time to invest into it right now.
Back then when I developed DashCache, I used Caffeine simulator to compare it to dozens other heuristics implemented in Caffeine using real-world data (twitter traces) and it was better than most of then (including LFU). It lost to algorithms that keep some information about evicted items (like TinyLFU or its extensions).
I agree. This is why DashCache is designed to be very efficient in terms of CPU and memory, while providing very strong adaptive capabilities for mixed workloads.
If think the most important part in this design is the realization that we do not need a global order in order to choose an entry to evict: other heuristics like LRU, LFU, frecency - they strive to provide a single number, a metric. Obviously this metric lies because a single number can not represent well both qualities like frequency and recency. DashCache, on the other hand, does not try to do it at all. Instead, it implicitly defines a partial order between entries in the same bucket or segment....