kuujo

I was cleaning out my GitHub repos the other day when I stumbled across this old gem I’d long forgotten, and I thought I’d share it. When I was at the Open Networking Foundation, I did some research developing a low-latency consensus algorithm that achieves consensus in a single round trip in the normal case using an SDN-enabled clock synchronization protocol to order changes, and falls back to a more traditional consensus protocol (largely derived from Viewstamped Replication) when network ordering fails. The protocol was inspired by NOPaxos and was an attempt to build upon that work by replacing its single-point-of-failure sequencer with a high precision clock synchronization protocol implemented with programmable network devices.

The repo is a TLA+ spec describing the JIT Paxos protocol. I feel it prudent to warn that this project was just an experiment to see if/how high-precision clock synchronization could enable new consensus algorithm. NOPaxos was an innovation in consensus with software-defined networks, and their work inspired me to see if I could take that innovation even further. IIRC the protocol still had at least one glaring performance issues when I left it (not to mention the obvious issue that low latency can only be maintained with low concurrency). But nevertheless, there seems to be some interest in this work on Twitter, so I thought I’d share it here as well.

Perhaps my work can inspire someone to think outside the box and take consensus to the next level, as NOPaxos did for me.

We’re not attempting to replace any of the existing Go logging frameworks. Dazl is an abstraction layer above structured loggers like slog. We could add a slog plugin (we have zap and zerolog plugins already). Essentially, what you would get from that is the ability to configure slog loggers (log levels, fields, outputs/writers, sampling, etc) via dazl configuration files.

But the fundamental role of dazl is really to eliminate dependencies on frameworks like slog as much as possible, particularly in Go libraries. If libraries use dazl rather than a framework like slog, users of the library can then import whatever logging backend they want to use (slog, zap, zerolog, etc). This becomes really useful when dazl is used by multiple libraries depended on by a Go application. Often different libraries will use different logging frameworks (if they even have logging at all), and this can produce inconsistent log messages with no unified interface for configuring loggers across your dependencies. In a perfect world, those libraries you depend on would use dazl for logging instead, enabling the top-level project that depends on those libraries to configure a single logging backend that’s shared across all the dependencies.

“Consisteny without clocks” seems like a misnomer considering Raft indexes are tantamount to a logical clock. Perhaps it should be “consistency without wall clocks.” But that’s not really novel by itself, except in the context of geo-distributed databases.

Copycat is a complete extensible implementation of the Raft consensus algorithm that was heavily influence by some of ZooKeeper's algorithms. You can read extensively about the Raft implementation in the architecture section of the docs. http://atomix.io/copycat/docs/

This is not totally true. There are ways to ensure a read from a follower sees the latest writes from the same client. You do this by sending the index of a client's write back to the client upon success and the client sends that index to a follower when reading. The follower awaits logs up to that index before servicing the read. Some Raft implementations do this, but clearly it can cause greater latency when switching between writes and reads and still only gets you sequential consistency and not linearizability.

I totally agree. I recently moved to LA for a job. A couple of months before moving I had bought a nice road bike. Even though I live 2.5 miles from work now, that road bike still sits in my closet every day. I would love to ride it to work, but LA doesn't even have enough lanes to support both driving and parking, let alone biking. The roads are full of traffic and impatient drivers and I value my life too much to risk riding my bike on the couple feet of shoulder that might exist on those roads.

I think the two business arguments for open source are:

Open sourcing software assists in its stability by creating a larger user base (users who report bugs) and attracting contributors (free development).

At the same time, open source contributions can promote a favorable view of the company as giving back to the community from which they presumably take. This can be attractive to prospective employees. I've been fortunate enough to be able to work on open source projects and indeed contribute them back to the open source community, and I likely would not be interested in any position where that were not the case to some extent.

Of course, there is always a need for a lot of software to remain proprietary. But at the same time, there are plenty of cases where the benefits of the open source community (testing and contributions) outweigh the risks of helping your competitors (open source generally useful tools, not specific competitive advantages in your industry).

I have to say, as a young engineer working at a startup full of cutting-edge technologies, this seems to be the best advice I've seen yet (if that type of environment is what you're looking for).

The proposed optimizations are pretty interesting, but I think my favorite such optimization has been proposed by Ayende Rahien. As with the optimizations outlined in this paper, his is related to leader election.

One of the issues with leader election in Raft is the potential for split votes. Raft tries I guard against this by randomizing election timeouts in order to discourage two candidates from requesting votes at the same time. What Ayende suggests, though, is to add a pre-vote stage wherein followers poll other nodes to determine whether they can even win an election prior to actually transitioning to candidates and starting a new election. This ensures that only up-to-date followers ever become candidates and thus prevents nodes that will never win an election from ever transitioning to candidates in the first place.