News for hackers. Stuff that matters.
Re:A generalised solution to distributed consensus by ccleve on Saturday March 09, 2019 @04:58PM (#19348761)
To make it work with any kind of performance, all message passing needs to be asynchronous, and you need an event loop to process messages one at a time. Things are much simpler if your code is effectively single-threaded when processing an inbound message. That took me a while to figure out. Futures and promises are very helpful.

It took a while to get the building blocks together: the log file, which can be traversed, read, and rolled back/truncated (use a kv store for this, trust me), and the transport mechanism to deal with sending complex objects and handling errors and timeouts.

Tracking terms, states (leader, follower, candidate), rejecting out of date terms, voting, elections -- it's all very complex. You'll have to build some really, really good multi-threaded tests that simulate out-of-order and dropped messages, then really pound away at the system. And good logging to find that odd corner case that keeps coming up and throwing mysterious messages.

I haven't even tried membership changes yet.

Re:A generalised solution to distributed consensus by bch on Saturday March 09, 2019 @09:16PM (#19349785)
> use a kv store for this, trust me

Order is important here - is this inherent in your key?

Re:A generalised solution to distributed consensus by ccleve on Saturday March 09, 2019 @10:48PM (#19350063)
Yes.

I did two implementations, one with plain text files and one with a kv store. For plain text files, I had to maintain file offsets of the records so I could traverse backward and truncate records that had to be replaced, and then break the files into segments so I could retire segments that are no longer needed. It had all kinds of little bugs.

Then I reimplemented using a kv store, and it took a tiny fraction of the time. And surprisingly had better performance.

Just create a good interface at the start, use the kv store underneath, and then later if you don't want the dependency swap it out for the more complicated code.

Re:A generalised solution to distributed consensus by bch on Saturday March 09, 2019 @11:45PM (#19350333)
Cool - I’d like to talk about your design and how you arrived at it if you’re interested.
Re:A generalised solution to distributed consensus by ccleve on Monday March 11, 2019 @11:56AM (#19359903)
Happy to. I just added my email address to my profile.