News for hackers. Stuff that matters.
Re:IceFireDB: Distributed disk storage database based on Raf... by tyingq on Saturday August 21, 2021 @10:41AM (#28257385)

  SET: 253232.12 requests per second
  GET: 2130875.50 requests per second
The 10:1 throughput ratio for GET vs SET is interesting. Redis being in-memory, the rates there are pretty close to the same for read/write.

Is a 10:1 ratio typical for a storage backed distributed kv store?

Edit: Looks like CockroachDb has roughly a 3:1 ratio, similar for YugabyteDB:

https://www.cockroachlabs.com/docs/stable/performance.html

https://forum.yugabyte.com/t/large-cluster-perf-1-25-nodes/5...

Also ~3:1 for etcd:

https://etcd.io/docs/v3.4/op-guide/performance/

Re:IceFireDB: Distributed disk storage database based on Raf... by jandrewrogers on Saturday August 21, 2021 @12:52PM (#28258472)
Comparisons of read/write ratios has to account for several differences in design and implementation. Representative benchmarks are difficult.

Things that can make a difference: Databases have subtly different definitions of "durability", so they aren't always doing semantically equivalent operations. Write throughput sometimes scales with the number of clients and it is not possible to saturate the server with a single client due to limitations of the client protocol, so single client benchmarks are misleading. Some databases allow read and write operations to be pipelined; in these implementations it is possible for write performance to sometimes exceed read performance.

For open source databases in particular, read and write throughput is significantly throttled by poor storage engine performance, so the ratio of read/write performance is almost arbitrary. That 3:1 ratio isn't a good heuristic because the absolute values in these cases could be much higher. A more optimal design would offer integer factor throughput improvements for both reading and writing, but it is difficult to estimate what the ratio "should" be on a given server absent a database engine that can really drive the hardware.

Re:IceFireDB: Distributed disk storage database based on Raf... by sitkack on Sunday August 22, 2021 @10:32AM (#28265778)
It depends, yes but ... (not discounting any of the above).

One sees a lot of 3:1 in practice due to the replication factor. If you have 3 copies of the data and the client can read from any node, you get 3x the read performance as having to have a quorum write on two out of three nodes.

To the GP, for a rough swag of what is possible out of given hardware, a combination of FIO and ACT (measures IO latency under a fixed load) is a good start.

https://fio.readthedocs.io/en/latest/fio_doc.html

https://github.com/aerospike/act

Re:IceFireDB: Distributed disk storage database based on Raf... by bob1029 on Saturday August 21, 2021 @11:18AM (#28257667)
> Is a 10:1 ratio typical for a storage backed distributed kv store?

In a single-node system, the best way to increase your write throughput is to batch requests over small chunks of time. Ultimately, the amount of writes you can perform per unit time is either bounded by the underlying I/O sequential throughput, or the business constraints regarding maximum allowable request latency. In the most trivial case, you are writing a buffer containing the entire day's work to disk in 1 shot while everyone sleeps. Imagine how fast that could be.

A distributed system has all of the same properties, but then you have to put this over a denominator that additionally factors in the number of nodes and the latency between all participants. A single node is always going to give you the most throughput when talking about 1 serial narrative of events wherein any degree of contention is expected.

Re:IceFireDB: Distributed disk storage database based on Raf... by refenestrator on Saturday August 21, 2021 @10:53AM (#28257483)
Raft involves waiting for fsync on a majority of nodes, so that's not too surprising.

'Typical' is a matter of what guarantees you want to give.

Re:IceFireDB: Distributed disk storage database based on Raf... by toolz on Saturday August 21, 2021 @11:28AM (#28257756)
Typically people use raft for leader election which in turn can coordinate writes. I don't think the writes are being fsync'd in the raft logs here. At least I wouldn't expect that behavior.
Re:IceFireDB: Distributed disk storage database based on Raf... by skyde on Saturday August 21, 2021 @03:30PM (#28259814)
For the raft algorithm to be correct Fsync is required on a majority of node otherwise you are technically not implementing Raft.

The reason is that in Raft if a node acknowledges to the leader that it wrote something to the log it must not later accept a different write in the same log position.

This mean if for some reason server rebooted with dirty buffered writes that could not be flushed in time. it’s supposed to forgot everything it know and rejoin the cluster using a brand new node id.

Re:IceFireDB: Distributed disk storage database based on Raf... by alexchamberlain on Saturday August 21, 2021 @01:33PM (#28258858)
Each write should be fsync'd to the WAL, right?
Re:IceFireDB: Distributed disk storage database based on Raf... by stingraycharles on Saturday August 21, 2021 @04:02PM (#28260068)
Yes but those can happen at the convenience of the particular node, not necessarily as a globally chrckpointed fsync()
Re:IceFireDB: Distributed disk storage database based on Raf... by refenestrator on Saturday August 21, 2021 @04:19PM (#28260230)
Then you're sacrificing consistency guarantees. If less than a majority have committed a write, it could be lost while the cluster still has a quorum up.

Waiting to report success until a majority have committed allows you to make guarantees with a straight face.. "it will probably be committed in the near future" is not the same thing.

Re:IceFireDB: Distributed disk storage database based on Raf... by rubiquity on Saturday August 21, 2021 @08:07PM (#28261755)
Not necessarily. As long as your quorums always overlap you can have weaker commit requirements. See the FPaxos paper and some of Heidi Howard’s blogs for more on this.

https://fpaxos.github.io/

Re:IceFireDB: Distributed disk storage database based on Raf... by jimsimmons on Sunday August 22, 2021 @02:33AM (#28263397)
FPaxos requires commit quorum and subsequent promise request quorum to intersect. It doesn't bring any significant thruput benefits
Re:IceFireDB: Distributed disk storage database based on Raf... by rubiquity on Sunday August 22, 2021 @11:50AM (#28266447)
I didn't say FPaxos brought throughput benefits. I pointed out that you can still have a correct implementation without requiring majority quorums.

But since you brought it up, FPaxos can have both lower latency and higher throughput than MultiPaxos/Raft[0] ;)

0 - https://dl.acm.org/doi/pdf/10.1145/3299869.3319893

Re:IceFireDB: Distributed disk storage database based on Raf... by jimsimmons on Sunday August 22, 2021 @06:00PM (#28270079)
Great link. Need to look into what WPaxos is.
Re:IceFireDB: Distributed disk storage database based on Raf... by jimsimmons on Saturday August 21, 2021 @02:42PM (#28259386)
You don't understand Raft. Quorum has to fsync for commit
Re:IceFireDB: Distributed disk storage database based on Raf... by rubiquity on Saturday August 21, 2021 @08:19PM (#28261814)
It reads like you and GP are talking about two different things. You’re talking about Raft specifics and the GP seems to be talking about a Vertical Paxos like setup where Raft is used for configuration and the data path uses another replication algorithm such as Primary-Backup or Chain Replication.
Re:IceFireDB: Distributed disk storage database based on Raf... by jimsimmons on Sunday August 22, 2021 @02:35AM (#28263409)
Hmmm that sorta makes sense I guess. Sorta because raft is a replication algorithm. If you don't use raft in datapath you don't get any of its guarantees
Re:IceFireDB: Distributed disk storage database based on Raf... by rubiquity on Sunday August 22, 2021 @11:53AM (#28266468)
Most data paths don't need consensus for replication. You can implement ACID Transactions on top of many other replication algorithms for example. Ultimately the choice comes down to read/write ratios. Chain Replication has much better read throughout than Paxos and Raft.
Re:IceFireDB: Distributed disk storage database based on Raf... by jimsimmons on Sunday August 22, 2021 @06:02PM (#28270095)
I see. What's the catch though. Sounds like free lunch. Is there some gotcha with partition tolerance?
Re:IceFireDB: Distributed disk storage database based on Raf... by rubiquity on Sunday August 22, 2021 @09:24PM (#28271437)
It’s the case of 2F+1 versus F+1. Paxos/Raft offer fault tolerance where as other replication algorithms don’t. If you have a node failure in Primary-Backup or Chain for example you need to reconfigure before committing more writes. However in practice and certain environments reconfiguration can be faster than or the same as a leader failure in Paxos that requires running Phase 1 again.
Re:IceFireDB: Distributed disk storage database based on Raf... by jimsimmons on Monday August 23, 2021 @02:37PM (#28280102)
Got it, makes sense. To the extent this can be formalized, I feel this is a much better alternative than consensus based approaches
Re:IceFireDB: Distributed disk storage database based on Raf... by inglor on Saturday August 21, 2021 @10:53AM (#28257485)
Often reads of data already committed only need to hit one node but writes need to wait for a majority so they need to wait for multiple nodes to receive and acknowledge the write.

I haven't checked the code though so I might be off.

Re:IceFireDB: Distributed disk storage database based on Raf... by AlphaSite on Saturday August 21, 2021 @01:53PM (#28259013)
I wonder how geode performs here.