Benchmarks done on only one metric are often misleading (and in commercial circumstances usually intentionally!). Would love to see visually what trade offs Dragonfly is making to achieve the numbers from the chart at the top of the README. If excellent technical work means there really is no trade off, that would also be a great reason to chart it visually.

Also as a Redis replacement, it's not clear what durability is offered, and for most Redis use cases this is close to the first question

How you would suggest to demonstrate visually that there are no trade-offs?

The tradeoff the way I see it - one needs to implement 200 Redis commands from scratch. Besides, I think DF has a marginally higher 50th percentile latency. Say, if Redis has 0.3ms for 50th percentile, DF can have 0.4ms because it uses message passing for inter-thread communication. 99th percentiles are better in DF for the same throughput because DF uses more cpu power which reduces variance under load.

Re-durability - what durability is offerred by Redis? AOF ? We will provide similar durability guarantees with better performance than AOF. We already provide snapshotting that can be 30-50 faster than of Redis.

There is usually a tradeoff between latency and throughput, although I'm not so sure this would be true for your innovation, since you've eliminated a whole chunk of fixed overhead from the system (syscall batching). However, batching in general often implies added latency

If I recall ScyllaDB has some excellent examples of demonstrating this particular tradeoff visually. A simple option would be a scatter plot where X = latency, Y = load or similar, with points coloured according to the system under test. Probably there is a better option, but this would likely be enough to sell me at least

Well in that case, I hope to have an good answer for you. Right below the benchmark graph I deliberately put a table that shows 99th percentile lantencies at *peak* throughput as reported by memtier_benchmark. 99th latency percentile of Dragonfly at its peak throughput. I put it here as well

  op r6g c6gn c7g
  set 0.8ms 1ms 1ms
  get 0.9ms 0.9ms 0.8ms
  setex 0.9ms 1.1ms 1.3ms

There are also things that are unmitigated improvements.

Does Dragonfly have a timeseries extension? Or does it support extensions?

It does not. If you refer to Redis Modules we are not planning to support those - we can just implement same features within DF.

Presumably, most of the performance benefit comes from using linux io_uring, which limits it to using recent linux kernels.

Of course, it is also possible there are situations where it doesn't perform as well.

Tradeoff is inability to run on anything besides recent Linux kernels

Does anyone run production Redis/memcached outside Linux any case?

Redis/memcached are used in production on FreeBSD servers (and other BSD systems).

Probably not many in production, but the "recent kernel" is going to be the more important part.

I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all.

Note: I never use kernel modules due to the issues rhel/debian and I have had with such things in the distant past.

> I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all.

I'm not sure that concern is justified. It seems io_uring was pushed as part of the 5.1 linux kernel release, and Ubuntu 20.04 LTS seems to have been shipped with 5.4.

https://packages.ubuntu.com/search?keywords=linux-image-gene...

Also, a quick Google search pointed to io_uring patches for Ubuntu 18.04 LTS.

The kernel has its own LTS releases, and running Linux in the cloud is a well tested use case. Any library not available in your distro of choice can be built from source and tracked manually, which usually works well enough.

I suppose because of io_uring?

It does sound like extendible hashing might have downsides in some scenarios also.

Sad that you can't run dragonflydb on DragonFlyBSD.

Thanks for your work.

Curious: Why BSL? Why not open core [0] (or xGPLv3) like what most other commercial OSS projects seem to be doing?

[0] https://archive.is/nT9DF#selection-410.0-410.1

This BSL Business Source License is endorsed by neither the Free Software Foundation nor OSI

https://www.gnu.org/licenses/license-list.en.html

https://opensource.org/licenses/alphabetical

That's a big red flag.

I'll be interested to see where the community lands on this years from now. It's frustrating to see so many databases relicense under licenses that are almost OSS but not quite. Yet it's totally understandable - it's hard to pay database engineers when you give your product away and then AWS neuters your hosting and support business.

I just wish more developers could follow Grafana's path and switch to the AGPL when that becomes a problem, instead of switching to fauxpen source licenses.

I think its ok that we have innovation at legal front as well. And innovation requires exploration, choosing road not taken...

> I think its ok that we have innovation at legal front as well. And innovation requires exploration, choosing road not taken...

Chosing to release software under a non-FLOSS, proprietary license is hardly innovative, and I'm afraid trying to frame it that way sounds like you want to have your FLOSS cake and eat it too.

The innovation should take place within the FOSS space, though.

Why? FOSS is, at least initially, a charitable project. You work on a database for months then give it in its entirety away for free to all your potential customers and even your competitors with better established sales teams (e.g. Amazon).

They are explicitly not a charity. Isn’t it okay to be a for profit company that sells software with a closed copyrighted source?

Red Hat isn't a charity either. They make a nice profit, and all their stuff is FOSS.

Red Hat makes a Linux distribution. It was not their choice whether to be charitable or not, they are the recipients of the initial charity of the Linux founders and are bound by contract to continuing making their code open as a price for building on open code.

If Red Hat could change the license, would they? Maybe. (Not now but you can’t say the leadership won’t make that decision during financial crisis).

But there's more to a Linux distribution than just the kernel, and a lot of the other pieces aren't copyleft, so Red Hat can change the license, but they choose to keep them open source anyway.

No, it's a collaborative project initially. You work on a tool for months with other developers that each have their own employer, but all benefit from the sum of the work.

That there's freeloaders that benefit from the work without having to expend labour is a side-effect, not the primary motivation.

The AGPL is not without its own controversies and some would lump that under fauxpen source licenses too.

What controversies, and who would? The FSF (obviously), the OSI, the DFSG, and Fedora are all okay with it.

yep it's a big red NO for me for sure

This looks quite cool from a technical perspective, but the unusual license definitely gives me pause. Largely because I'm not familiar with BSL. Others say it's increasingly popular, but with a little googling the acronym is still somewhat ambiguous - opensource.org lists BSL as the "Boost Software License" which looks more like BSD. This kind of confusion doesn't support the idea that this is a solid trustworthy OSS license.

Still, I really appreciate that you didn't choose a copy-left license.

On the license front, what is the "change license" clause listed? It says something about changing in 5 years. Does this mean it will become Apache licensed in 2027? Why would you put that in there?

It's exactly that. It gives us a little chance to fight against the Giants.

In 5 years the initial version becomes Apache 2.0 then the next version and so on and so forth. CockroachDB uses similar license. MariaDB uses that, Redpanda Data and others. You are right that acronym is confusing - it's not Boost license, it's Business License. Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.

I certainly don't think you should pick a permissive license, and find yourself under competition from companies running your own software.

However, I'm sad that instead of going with an Open Source license that protects against that, you're using a proprietary license. That alone is a nonstarter for many users, not because they want to compete with you but because they want to protect themselves and make sure they have a firm foundation to build on.

Much of the software you're citing as examples moved from Open Source to proprietary, harming their users in the process, and causing many users to seek alternatives.

Do you think AGPL offers such protection? I think even AGPL is fine for Amazon et al, they are happy to dump the code somewhere while selling a managed service with enough wrapper layers. But AGPL is toxic also for self-hosters who aren’t trying to build a managed hosting service, since it applies equally against all parties.

> AGPL is toxic also for self-hosters who aren’t trying to build a managed hosting service

Absolutely not. First of all, you can always use any software under AGPL as released without any limitation.

Secondly, if you were to make changes and release them you can still run it any way you want.

Thirdly, if you are making changes for internal use, and for some reasons your really want to keep them secret, you can STILL run the database to your heart's contents as long as you don't provide it as a service to the public.

AGPL is much more permissive than people think. It's just providing a degree of protection to developers and and users from patent trolls and other uncooperative entities.

Exactly - the grandparent is asking “can a true copy left / Free Software license protect a young startup from Amazon reselling their software”, and my assertion is no, it cant.

AGPL strikes a balance between the needs of users, upstream and intermediate developers.

Here [A]GPL is Schrödinger's license. At the same time people complain that it's too restrictive and not restrictive enough.

If you want to completely prohibit any SaaS usage closed source is the only option, like BSL. And I'll stay away from your software.

Can you give an example of an AGPL product being resold by Amazon or someone like them?

MongoDB is the most famous example to transition from AGPL to a closed license. MongoDB was AGPL until October 16, 2018. Since October 16, 2018, it uses Server Side Public License (SSPL), which they authored.

From their FAQ:

> The market is quickly moving to consume most software as a service. This is a time of incredible opportunity for open source projects, with the potential to foster a new wave of great open source server side software. The reality, however, is that once an open source project becomes interesting, it is too easy for large cloud vendors to capture all the value but contribute nothing back to the community.

HN discussion from 2018: https://news.ycombinator.com/item?id=18229452

FAQ: https://www.mongodb.com/licensing/server-side-public-license...

Today, MongoDB is an AWS partner: https://aws.amazon.com/quickstart/architecture/mongodb/

The most recent notable example of open-source software going closed because of Amazon specifically is Elasticsearch, which switched from Apache to SSPL. The announcement blog post was titled "Amazon: NOT OK - why we had to change Elastic licensing".

HN discussion from 2021: https://news.ycombinator.com/item?id=25833781

FAQ: https://www.elastic.co/pricing/faq/licensing

MongoDB is definitely an example. Did switching licenses actually "fix" their "problem" though? And ElasticSearch doesn't count since it was a pushover license before and not the AGPL.

Mastodon is AGPL and there are various cloud hosting providers.

There really aren't that many popular AGPL server software to start with though, so it's an unfairly biased question.

AGPL does not require source distribution of your application, unless the AGPL code is accessed directly by your users.

So, if this were AGPL and I have an closed webservice that uses this, it is in the clear.

But then Amazon is free to host this as a paid service and take all the money.

Redis is doing just fine, PostgreSQL is doing just fine. They are both real open source.

The performance improvements are not worth the legal/compliance overhead of adding non-FOSS to my stack. Much less so if the performance improvements are due just to some optimization choice in the underlying system. In the next 5 years, it will be easier to have Redis adding the io_ring optimizations than for this new project to become uniquely better.

Now I’m curious where you work and how compliance requires FOSS!

You misunderstood. What I am saying is that is precisely because I am working on projects of different sizes and potentially with different customers, I do not want to have the burden of having to check if the projects have specific compliance requirements.

I have a SaaS [0], my own (AGPL, by the way) open source project [1] and also work have the occasional contract job. It's much easier to say "just use Redis because it is FOSS and it gets the job done" across the board then to try to special-case the tools based on licensing requirements.

[0]: https://communick.com

[1]: https://hub20.io

And? AGPL is not a non-commercial license. If Amazon wanted to provide such a service, they would certainly be permitted to. As long as they provided any changes they make to the the AGPL source code. The AGPL would not "infect" their hosting infrastructure.

We do not choose AGPL so we would not harm our users. Lets not confuse the means and the goal here. AGPL is copyleft and restricting. Fair users refuse using it. OSI does not help technological companies to find a fair solution and force us to go outside of "OSS".

> We do not choose AGPL so we would not harm our users.

This is a first. If anything, it seems clear that a FLOSS license such as AGPL achieves exactly the opposite: is highly protective of the users' best interests, both in the short term and long term.

Could you elaborate on why do you think that users are harmed by standard, run-of-the-mill FLOSS licenses?

Depends on who you consider 'the users'. If you are a 'user' in the sense of an individual working on their own stuff, then yes, AGPL protects you a lot.

However, if you are a 'user' in the sense of a company wanting to make software to sell as a service, it doesn't protect you very much, and can make you vulnerable.

> However, if you are a 'user' in the sense of a company wanting to make software to sell as a service (...)

That does not meet the definition of "user", does it?

In addition, if that was the motivation behind that absurd claim, why not be honest and just say "we don't release our software under a FLOSS license because what we actually want is to keep it proprietary"?

It means ‘user’ in the sense of the person who is choosing what software to include in their work stack, and most software of this type is going to be used in a professional setting. If you use a license that most legal departments won’t let their developers use, your software won’t gain much adoption… which is especially important if you are planning on making money from your software. People aren’t going to be paying for support for their personal projects.

How does AGPL can make a company trying to sell a software* as service vulnerable?

*Assuming it's not a memory-store as a service, obviously.

I don’t know, but I do know my legal department says we can’t use it, and I have heard the same about many other companies.

Most legal departments don't allow for copyleft licenses. This is because most legal departments don't actually make their own legal determinations but instead rely on industry norms. And google set the stage here ages ago with their no-GPL (and especially, no-AGPL) policy.

However this is rather uniquely due to Google's monorepo infrastructure. Google quite literally has all its products in the same source code repository, being built and statically linked together to create a single system binary (or so I am told--I've never worked there). In this case AGPL would virally infect the rest of your code and you could find yourself in trouble. Even if they don't, they might get hit with a discovery request for the source code of some project to prove compliance, and in the process have to provide their entire monorepo and all its business secrets to the court. Etc. Etc.

But these are concerns which stem directly from Google's monorepo architecture. It's not generally true of the whole industry. And yet Google's fear of copyleft--rational though it may be in their particular instance--has been copied throughout the entire industry. 99% of the companies out there have nothing to fear from the AGPL. But hey, if you're some random in-house lawyer at company XYZ, who are you to question Google's legal precedent?

Using a non-OSI approved license hurts your users considerably more than AGPL (and this is coming from someone who generally doesn't like the AGPL). Legal teams are far less likely to approve licenses like the one you've chosen, than AGPL.

I could convince a legal team to allow an AGPL service, depending on how we were using it. It would be difficult to get them to allow us to use this license, and I definitely couldn't convince them to allow us to contribute to software using this license.

Effectively, you've ensured you won't have a community, because in reality the project is closed source, and dead if your company dies, which means it's not an option for me to consider.

This is not representative of my experience. AGPL is unconditionally banned at most companies where I know the answer to this question. They don't care if the license is OSI-approved, they care if it aligns with their business requirements and objectives. Most companies are not ideological about open source.

BSL is at least making an attempt to address the legitimate concerns of businesses -- both sides -- in a balanced and pragmatic way. I have no dog in this fight, being neither a BSL licensor nor licensee (nor AGPL for that matter), but having been party to the legal conversations at large companies I understand why BSL is generally considered to be more acceptable than AGPL. BSL may or may not be the right license for this software, I have no opinion, but businesses don't care about arguments from ideological purity. If BSL satisfies their requirements better than AGPL, and anecdotally all the evidence seems to support this notion, then they will go with BSL software.

To put it another way, if OSI is serious about being in the conversation about the future of software licensing, they must address the reasons so many companies refuse to adopt AGPL software.

I wonder how involved you've been in these kinds of discussions. I've run open source programs at companies, and worked together with the legal teams in terms of what was and was not allowed.

Transferring copyright to another company, without a contract with that company, for the most part is something a legal team isn't going to agree to. BSL only addresses one side's concerns, and it's not in your favor.

AGPL, when used for internal services that customers don't interact with, is generally not that difficult to get approved. How many companies are using mongodb, for instance?

I do not have any practical, first hand experience with either license. Re -agpl, i know for a fact that it is banned in corporations. We choose BSL to avoid this. Say if a team in amazon would want to use DF internally they could. I know for a fact that AGPL is a big no there. If we are mistaken and BSL is also banned then we will need to see what to do.

I hope people will understand the reason behind bsl. I hope people will see apache there and will help us to grow. Based on HN comments some already do.

As someone building software for a living I fully agree with your decision to not go full OSS. It’s always a hefty discussion but I feel most of the people only exist on the receiving end of software and do not have to worry how a license impacts real life distribution. Big corp (I’m from Germany) DOES hesitate to go with any form of non-Apache-2 licensed OSS software. In Germany often times OSS is even discouraged if there’s no commercial support for it.

I understand the emotions people have regarding licenses that do allow them to work with software as they please - but I also think it’s useless to argue around licensing of a software instead of focusing on the good it can do. Something being proprietary has NO impact on your choice to buy it when outside of the software world. Why does it here?

The problem you are pointing out is not the open source license, it's the lack of support. If they offered AGPL with a dual-license version and made a strong message about commercial support, they could please both crowds without compromises.

> I do not have any practical, first hand experience with either license.

I do have practical first hand experience with both. I can get a legal team to approve the use of AGPL, but I cannot get them to approve BSL, unless we've signed a contract with the company.

Having an understanding of the license, and some first hand experience seems like it would be a pretty important thing, when choosing it, especially for a business.

I don't want to enter in the merit of your company's choice, it's definitely up to you what license to use, but I'm confused about that statement around AGPL. How would AGPL harm users?

By preventing them from using dragonfly. Many companies prohibit from dev teams(aka our users) from downloading and using agpl licensed software.

The lawyers often ban AGPL at tech cos

Which means they may pay for an alternative to the AGPL.

More likely they'll ignore the s/w. Huge amount of talent is working in such companies. Using AGPL will practically exclude them from interacting with the s/w.

Also, smaller companies like to play safe. If FAANG with their million lawyers aren't touching it, why should I take the risk.

Because unlike FAANG you don’t have millions of engineers to create the software from scratch. Companies pay for databases (Oracle, Gemstone, MsSQL, anything at Amazon). Almost all the risks (price changes, maintenance costs, etc.) can be managed adequately via a long term enterprise contract.

Yes, we’re going through this right now

If that's your only concern, then why not dual license? Give your users the choice whether they want to follow the current license or the AGPL.

We may do it in the future. We will see if the community will accept Dragonfly with BSL. MariaDb were pretty famous and i do not remember any loud objections to their license.

I was quite excited to see Dragonfly. However now that the license has been pointed out, I won't be touching it with a 10 foot pole. I wish you the best of luck, but I'll never use it.

If it was AGPL I'd have been a happy camper though.

There were a lot of loud objections to their license, and remember that MariaDB the database isn't licensed that way (it's GPL), just the much less used MaxScale and CMAPI.

> Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.

No, there are plenty that still use permissive licenses.

GitLab uses MIT and a custom license for EE: https://docs.gitlab.com/ee/development/licensing.html

Deno uses an MIT license and has some secret sauce that is currently just in hosted services AFAIK: https://github.com/denoland/deno/blob/main/LICENSE.md

PlanetScale has hosted services and an open source tool called Vitess which is Apache licensed: https://planetscale.com/ https://github.com/vitessio/vitess

Finally Redis has a BSD licensed core, a source available license for additional modules, and a closed source license for enterprise. https://redis.com/legal/licenses/

Redis has a BSD license because it was written by antirez who did not intend to make it a commercial enterprise. Today its too late to change and wont help Redis Inc to protect their ip. We looked at elastic, mongodb, cocroach labs, mariadb, redpanda, atlassian, grafana labs , scylladb. The last two are agpl but the motivation is the same: to protect themselves.

Bravo for picking the BSL and writing your license the way you have. It shows maturity in that you and your company know what they are doing and intend to be there for the long haul.

MariaDB is GPL there are some enterprise components which are BSL.

Was excited to see the project but now seeing it is not Open Source it means 1/10th of value

They probably don't want to end up being replaced by an AWS/GCP/Azure service. In my opinion, the BSL is a fair license model, especially if it is limited in duration (let's say 2-3 years BSL then automatically changing to Apache/BSD).

A duration limit in the license, after which it becomes a permissive license, seems the critical point.

Accomplishes the goal of preventing a cloud provider from stealing customers, but also ensures customers don't get caught in an "always tomorrow" trap when the deadline comes and the company realizes it only hurts them to fully share it.

Seems to align all interests pretty nicely.

(I'm as big of an OSS supporter as anyone, but we can't pretend we still live in a time where Google / Amazon / modern-Microsoft don't exist)

Exactly. Finding the right balance is the right thing to do. I am not angry at cloud providers , but i am angry at OSI and their blind idealism that leaves us out there alone. I think that not solving the problem by them, hurt first of all the oss community.

> I think that not solving the problem by them, hurt first of all the oss community.

F/OSS means a very specific thing. If one can't possibly build a rocketship business, that isn't F/OSS fault. Of course, you've got an alt-movement in response to OSI and FSF's rigid adherence to its principles, speared on by tech companies who (think they) got burnt by other tech companies.

See also: https://news.ycombinator.com/item?id=25896162

I think romange put it well: balance.

There's a lot of room between {completely F/OSS, that a cloud provider can implement and bankrupt the company supporting project development} and {completely evil closed source company like Oracle}.

In the end, we all want good software, with the maximum amount of permissions and source, free. It's just a question of tweaking the support model to get there.

IMHO, the "Don't call yourself OSS if you're not 100% OSI OSS" is counter-productive. But I understand why they do it, and I understand the history and abuses that caused them to do it. I'd just say if we no-true-Scotsman our approach in a way that precludes profitable, sustainable OSS companies... we're going to have less quality OSS. :(

But my entire stack is already in AWS. One service provider deciding they don’t want AWS making a service out of it just means I have to make the effort of self-hosting it, not that I’ll suddenly end up using their service (which is outside my VPC).

can you clarify why you feel this way, plenty of companies use confluente Kafka or mongodb with VPC peering? Would AWS private link make you feel better about network security?

GPL is copyleft and more restrictive in some elements. BSL 1.1 is actually quite popular nowdays.

Apologies if this was already asked -- I saw you guys already showed benchmarks in AWS itself, but would it make sense to have some benchmarks outside of AWS (or run benchmarks in a VM with Redis/DF/Keydb installed within)?

I ask this because I'm unsure if AWS Redis has any modifications on top of the Redis software itself, which would affect the speed, or even make it a bit slower. For example I know MS Azure's version of Redis restricted certain commands, and from a quick search AWS does something similar: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...

(edit: added "affect the speed" for clarity)

We compared DF with Redis OSS and KeyDB OSS, we have not used any of managed services for the comparison. AWS was used as a platform for using EC2 and cloud environment.

Two things:

Where's the benchmark compared to memcached?

Several years ago there was memcachedb, which could flush stuff to disk. While this operation was expensive, it was also useful, because you could restart instances without being overwhelmed by missing keys (data).

For the latter: your application quickly grinds to a halt if you need to build your cache from ground up after some kind of crash. This is a deal-breaker for many.

I gotta say, I looked through some files. I wish my code was this clean. its some of the prettiest c++ I've ever seen

When Jens Axboe praises you for your work [0], then it is a given you're at the top of your game.

[0] https://twitter.com/axboe/status/1531371740403617792

Indeed :) The moment i learned about io_uring I got hooked up immediately. People like Jens, Pavel are the type of innovators that make sure Linux stays on top of its game for the next generations.

I dunno... I took a look at a single file and found:

        static constexpr unsigned NUM_SLOTS = Policy::kSlotNum;
        static constexpr unsigned BUCKET_CNT = Policy::kBucketNum;
        static constexpr unsigned STASH_BUCKET_NUM = Policy::kStashBucketNum;

NUM_, _CNT, _NUM, three different prefix/suffix for what seems to me like the same concept. That just tickled my inner nit-picker.

You are welcomed to send a PR with the fix :)

Everyone has his dirty laundry :(

that's so minor

omg, can I marry you?

btw, it's not just pretty - it has brains too. take it for a spin.

is there an elixir client library?

I presume you just use a memcache or redis client? So yes?

From my experience, Redis performance seems to be all over the place depending on the circumstances. In what cases does this solution perform well and where does it fail? Most caches I worked with really love to have everything close by at low latency and works best when the consumers have about as much memory as the cache in the first place.

Did you think about re-using redis open source and "just" changing storage/replication? (like yugabytedb does with postgresql).

Why not reuse seastar framework?

Can you describe your distributed log thing? Is it like facebook-logdevice or apache-bookeeper?

I honestly think it's impossible to reuse Redis OSS to make it multi-threaded besides what KeyDB did. Btw, I did reuse some of the code - but it's around single value data-structures like zset, hset etc.

With multi-threading you need to think about all things holistically. How you handle backpressure, how you do snapshotting. How you implement multi-key operations or blocking transactions. So you need special algorithms to provide atomicity, you need fibers/coroutines to be able to block your calling context yet unblock the cpu for other tasks etc. All this was designed bottom up from scratch. Seastar could work theoretically but I am not a fan of coding style with futures and continuations - they are pretty confusing, especially in C++. My choice was using fibers - which provide more natural way of writing code.

I have not designed the distributed long thingy. Will do it in the next 2 months.

This looks like awesome work! I appreciate you operationalizing of some of the best things to come from computer science in the past decade or so.

Out of curiosity, are you discovering any new bottlenecks to performance outside of the software, given Dragonfly is able to process far more qps than most systems? I imagine the network and disk I/O could become stressed, but also I wonder if it breaks any assumptions of cross-core performance, hypervisors, etc. I know that cloud offerings typically mean that you can attach ginormous disk IOPS and NICs, but surely there are limits.

Yes, I found many different bottlenecks.

I was mostly running on AWS. In terms of hardware, for small-packets loadtests, most systems are constrained on throughput, i.e. number of packets per second. Some instances saturate on interrupts reaching 100% CPU on all cores and some can not even saturate the CPU and you will see that CPU is at 60% but you can not go beyond in throughput. The best systems network-wise are c6gn family types. They are also better than instances that other cloud provide. btw, you mentioned hypervisors... About 8 months ago I opened a bug on AWS Graviton team https://github.com/amzn/amzn-drivers/issues/195 - about performance issue they had on their instances at high throughput. Recently they issued the fix. I suspect it was in their hypervisor.

In terms of my software I found many performance bugs at those speeds. For example, using a default allocator is a big no. I use mimalloc for uncontended allocations. In general, you can not use mutexes and spinlocks at those speeds. Those will just cripple the system. Sometimes it can be very annoying since you can not rely on a 3rd party library without carefully analyzing its design. For example, I could not use openmetrics c++ library because it was not performant enough. Even to implement a simple counter, say to gather statistics for INFO command becomes an interesting engineering problem: With share nothing architecture, I use a lot of thread-local counters that I aggregate only when stats are pulled.

As a general note, I expect that Dragonfly will stay very performant with the tailwinds from recent hardware advancements. For example, c7g (Graviton 3) is much better than c6g and DF shows it.

I'm curious: since you're targeting Graviton, and AFAICT Graviton 2 and up support ARM LSE, have you tried directly using LSE for metrics? ARM LSE offers STADD to add to a 64-bit memory location without reading the contents or ordering the access. (I think LDADD with XZR as the destination is identical, but I could be missing some subtlety. I'm not an ARM expert.) You might be able to get good performance by using a global counter and just STADDing to it. x86 will perform terribly if you use XADD because it has no corresponding optimized forms.

Wow, its beyond my knowledge:) I use my software skills to write a performant software but rarely i go to the assembly level. And i have zero knowledge about specifics if each command on every architecture. The rare exception is a code in dashtable. the authors of the paper designed it to allow vectorizing of the find() operation.

Interesting. Have you considered/tried to use fdio/user space networking? In my experience, it greatly improves throughout (simple ip forwarding can be more than 10mpps per cortex a53 on some platforms). Fdio also has a so that you can preload in order to use its ip stack in your app (instead of Linux's).

See https://s3-docs.fd.io/vpp/22.06/developer/extras/vcl_ldprelo...

Have you thought about also offering dragonflydb as a library, like sqlite? This would avoid context switching when running everything on the same machine.

it could be done technically. not sure how widespread that usecase would be.

dragonfly is a linking of a library and a main file dfly_main.cc so without this file you will have the lib.

Redis is "Remote Dictionary Server". You gonna loose the remote part :)

Being able to use redis as an in-process L1 cache is something I’ve wanted to do. I don’t want the overhead of domain sockets.

This is what in-memory data grids can do, look at Apache Ignite for example: https://ignite.apache.org/

for development and demos it would help a lot, son one just runs the project at hand, instead of managing multiple procs

Is there any catch or gotcha for using this as a drop in replacement for Redis?

It's only at release 0.1 so I'd let them bake a little longer!

Looks awesome so far, though!

The license is different. Redis core is BSD 3 clause, which is a permissive open source license. Dragonflydb uses the Business Source License, with some additional use grants. Depending on your situation, that could be an impediment.

A lot less features. If you don't care about the missing features, then it might be a completely reasonable replacement.

I've been working on DF for the last 6 months and I implemented ~130 commands. It's still missing about 40% of the commands to reach parity with v5/v6.

I will continue working on DF. Primary/Secondary replication is my next milestone.

How apples to apples is the performance comparison given these missing commands?

Those commands wont affect the design of the df, hence they wont affect the performance. Its just work that needs to be done.

I can see it's missing some important commands like WATCH, and maybe the WAL for high durability contexts.

If you see that important command are missing, pks open an issue and we will prioritize their implementation.

Just wanted to say 'thanks' for doing such a great job responding to questions and comments. Nice job.

Thank you!

how many people have told you to rewrite it in rust so far?

Quite a few :) I wish i was 20 again - to have time to learn the language and to implement df in rust :) i am an old with old habits. Btw, I spent a short time coding in rust last year and i understand why the fuzz.

Any intention to implement Redis Streams?

+1 to this question

Guys, please open a bug in the repo and plus one it there :)

We plan to implement everything but your votes can affect the priority of the tasks.

curious about the consistency guarantees. redis is single threaded which seems old fashioned but guarantees that all transactions are serial, so there is no chance of conflict between reads, no issue with race conditions , a complicated problem to solve once concurrency is introduced . if the redis protocol is used how is this problem addressed?

Yes this problem is complex and i guess this is why it have not been solved before.

We do provide atomicity guarantees for all operations like Redis! We use an algorithm from a 2014 paper - see our readme, we provide the link to the paper.

Good stuff :-) I would also love to see a comparison against Scylla and Aerospike, even if the use-case seems slightly different?

I do not know about aerospike, have never run their software but Scylla are the champions in what they are doing. Having said that, Scylla provide full durability guarantees, that means they need to touch a disk upon each write or update. Disk I/O is relativle expensive in the cloud and requires either large instances or special storage optimized family types like i3 etc.

Long story short, I do think Dragonfly is the fastest in-memory database in terms of throughput and latency today. We will see if we manage to stay this way when we extend our capabilities with SSD tiering.

> that means they need to touch a disk upon each write or update

They don't fsync the wal on every write, it's probably done as group commit every x seconds or every x MB.

they need at least perform fdatasync before replying to a client, imho. so I do not think it's every x seconds. I am not deeply familiar with scylla so maybe I am mistaken.

I think you're wrong. It does that every x seconds. The disk writing is async and it's meant that you protect data-loss by keeping replicas in different availability zones.

interesting. i still need to learn a lot about how other systems provide durability guarantees. Based on what you are saying, Scylla guarantees H/A instead of full durability. I was thinking about WAL (AOF) for dragonfly and I kinda thought to start with "fdatasync once in every X seconds", but dragonfly is not a database. I would expect stronger guarantees from databases. Maybe it's inpractical to expect this in cloud environments...

It wouldn't have the performance that it has. Cassandra does the same. YugabyteDB does the same https://docs.yugabyte.com/preview/reference/configuration/yb....

> I would expect stronger guarantees from databases.

You can configure it and get lower performance.

> Maybe it's inpractical to expect this in cloud environments...

For benchmarks they use raid0 on local disks(not EBS) in aws vps.

Have any thoughts on Anna? They take a similar kind of shared-nothing approach, with CALM communication primitives.

Anna is a KV store, for those out of loop: https://muratbuffalo.blogspot.com/2022/04/anna-key-value-sto...

I remember that I read about Anna. Very interesting paper. From what I remember that require that the operations will be conflict free.

I think came to conclusion that it could be interesting as an independent (novel) store but not something that can implement Redis with its complicated multi-model API, transactions and blocking commands. I do not remember all the details though...

What prompted you to write DragonflyDB ? How did you know that Redis would consume more memory ?

I've wrote in in the background section of Readme.

Basically, I worked in a cloud company in a team that provided a managed service for Redis and Memcached. I witnessed lots of problems that our customers experienced due to scale problems of Redis. I knew that these problems are solveable but only if the whole system would be redesigned from scratch. At some point I decided to challenge the status quo, so I left the company and..and here we are.

Impressive to back yourself and leave your job to tackle such an interesting problem.

Yeah, some call it stupidity... Honestly, it's really irrational thing to do. My last company were good to me but i got the itch...

Sorry, missed your second question. Redis is using fork-based save for snapshotting and full replica sync. This means that memory pages are duplicated on write. That consumes more memory than DF implementation that implements algorithmic point-in-time snapshotting. In addition DF was designed to use less overhead memory in its keys table. As for the why, you can read about it here: https://github.com/dragonflydb/dragonfly#background

Very cool project! Are there any example projects using early stage Dragonflydb yet?

Is this a project of https://en.wikipedia.org/wiki/Atos ? How big is the team, how is it funded and suchlike?

no it's not us. we are two guys right now. bootstrapped with hopes that community will embrace the project and we will be able to grow the team.

What an impressive start to the project! Good luck, I hope you get to grow it in the ways you have planned.

btw, please follow me for updates about dragonflydb at https://twitter.com/romanger

I find the Redis benchmark suspect. Did you disable write to disk and live snapshotting? In production Redis shouldn’t be configured with write to disk.

Of course, I disabled. Why do you think it's suspicious? P.S. You can easily reproduce it - I wrote the command line and the instance type.

Yes

Do you know how DFDB compares, architecturally and performance-wise, to Pelikan and its different engines?

Not really but I read pelikan posts by twitter team.

One thing in common - we both thought that cache-based heuristics can be largely improved compared to memcached/redis implementations. We did it differently though. I think our cache design has academic novelty - I will write a separate post about it.

hey, great work. I couldn't find the specifics of the benchmark. Is there, by any chance you compare 1 instance of single threaded redis running on 64vcore to a multithreaded key-value store?

Can we see such disparity in benchmark even if we run Ncore instances of redis in parallel?

Thats what i did. I did not try to cheat. Running N redises is a completely different product. You will have n listening ports and your loadtest program will need to connect to each one of them. The power of dragonfly is that it hides all this complexity. You are saying that comparing single threaded redis to df is unfair, and i say - comparing n redises to df is unfair because you compare different products.

Do you have a saas cloud offering like redis cloud?

not yet. We are focused on building the community at this point.