Re:Dragonflydb – A modern replacement for Redis and Memcached
by romange on Monday May 30, 2022 @01:51PM
(#31561608)
Unfortunately unlikely. Shared nothing architecture works with messaging. So threads send each other messages like "give me the contents of that key" or "update this key". Operations like SET/GET will require a single message hop. Operations like RENAME require 2 hops. Transactions and Lua scripts will require number of hops as number of operations in the transactions. When it's in the same process - the latency penalty in negligible. But between different machines - you will feel it. But who knows, AWS already have very cool UDP-like protocol with very low latency... if this will become the standard inside cloud environments maybe we can create a truly distributed memory store that spans multiple machines transparently.
Re:Dragonflydb – A modern replacement for Redis and Memcached
by etaioinshrdlu on Monday May 30, 2022 @02:32PM
(#31562070)
The latency would be worse, but would the throughput still scale pretty much linearly? It might still be super useful.
Re:Dragonflydb – A modern replacement for Redis and Memcached
by romange on Monday May 30, 2022 @03:35PM
(#31562772)
Yes, VLL algorithm kinda solves the throughput problem. You will still have some issues with fault-taulerance - i.e. what you do if a machine does not answer you.
For intra-process framework it's not an issue (as long as we do not have deadlocks).
Re:Dragonflydb – A modern replacement for Redis and Memcached
by jayd16 on Monday May 30, 2022 @01:18PM
(#31561226)
It kind of sounds like simple sharing but across all cores in a machine. Anyone know if this is an accurate take?
Re:Dragonflydb – A modern replacement for Redis and Memcached
by romange on Monday May 30, 2022 @01:31PM
(#31561381)
It's not a simple sharing. It's actually share-nothing architecture and doing multi-key operations atomically is pretty complicated thing. I used paper from 2014 to solve this problem
Re:Dragonflydb – A modern replacement for Redis and Memcached
by jayd16 on Monday May 30, 2022 @01:54PM
(#31561657)
Maybe simple is the wrong word (perhaps I should have said familiar) but Redis sharding is similarly shared-nothing, is it not?
Re:Dragonflydb – A modern replacement for Redis and Memcached
by OPoncz on Monday May 30, 2022 @02:09PM
(#31561833)
In Redis cluster the client needs to be connected to all shards and manage those connections. Multi-key operation on different slots are not supported etc... Maintaining a cluster is not a fun responsibility. DF saves you in most cases from the need to grow horizontly which should be much simpler to maintain and work with.
Re:Dragonflydb – A modern replacement for Redis and Memcached
by jayd16 on Monday May 30, 2022 @02:19PM
(#31561918)
So is it architecturally different or does DF just have a much better UX?
Re:Dragonflydb – A modern replacement for Redis and Memcached
by romange on Tuesday May 31, 2022 @01:58AM
(#31566524)
it is architecturally different. DF has the orchestrator underneath. The general. The guy that handles all the incoming requests and makes sure they execute atomically. Redis cluster is an orchestra without the orchestrator. Everyone to himself. It's share-nothing that is exposed outside. To actually hide the complexity in a *transparent way* is a difficult problem. Researches write articles on how to solve this problem efficiently.
Could this architecture be extended to scale across multiple machines? What would be the benefits and costs of this?