_benedict

I can’t speak for the authors, but I have been lucky enough to be collaborating with them on behalf of the Apache Cassandra project, to refine and prove the correctness of the Accord protocol - a derivative of EPaxos we have integrated into the database.

It would be fantastic if such a project could be pursued for this variant, which has the distinction of being the only “real world” implementation.

Either way, thank you for the original EPaxos paper - it has been a privilege to convert its intuitions into a practical system.

“Paxos” is a term that can mean many different things, so it’s better not to get too attached to any one meaning especially in different contexts.

Multi Paxos is commonly used (especially in industry) as short hand for multi decree Paxos (in contrast to single decree Paxos), but “Paxos” most often refers to the family of protocols, all of which are typically implemented with a leader. It is confusing of course because single decree Paxos is used to implement EPaxos (and its derivatives).

It’s worth noting also that Lamport is (supposedly) on the record as having intended “Paxos” to refer to the protocol incorporating the leader optimisation.

I remember there being sufficient documentary evidence in the entrance/shop/museum bit to conclude it was most likely created by the very people who “discovered” it, to serve as a tourist attraction.

Thank you for linking the source material, unfortunately it badly contradicts you. It clearly shows that the _very first_ list of ten suggested search terms contained (pretty heavily) sexualised suggestions.

No, the ruling expressly refers to the list as non exhaustive, but given the other related references to misconduct (including the use of inappropriate language) it was not reasonable to infer that this example was gross misconduct.

Worth noting, not quite "everyone" does this. Cassandra uses "leaderless" (single decree) paxos, which has some advantages and some disadvantages (for instance, 1RT WAN reads from any region).

I agree with you that Paxos is simpler than Raft. The problem with Paxos IMO is that Lamport's original paper is impenetrable; lots of later writing is easier to understand, including those that describe more complex protocols. The intuitions are actually pretty straightforward, and transfer to all of the extensions to Paxos (which are not as straightforwardly compatible with Raft).

Raft may have helped more people get comfortable with distributed consensus, and sped its adoption, but being a sort of dangling branch of the tech tree I wonder if this may have stalled progress beyond it.

Not even a coincidence really, it's a very different kind of system. It's an implementation of Hermes with network layer integration. Hermes is designed with very different goals in mind, specifically within-DC consensus with minimal failures (with the caveat I am not intimately familiar):

- Every replica must acknowledge a write, which is undesirable in a WAN setting, due to having to wait for replies from the furthest region

- At most one concurrent "read-modify-write" operation may succeed, so peak throughput is limited by request latency

- Failure of any replica requires reconfiguration for any request to succeed (equivalent to leader election), so the leaderless property here does not improve tail latencies, indeed it is likely harmed by exposing your workload to more required reconfigurations

Cassandra is designed for multiple (usually quite far apart) DC deployments that want to maximise availability and minimise latency, and where failure is expected. Here a quorum system is typically preferable for request latency.

Revolutionary may be an overstatement, it just affords different system characteristics. There's plenty of literature on the topic though, starting generally with EPaxos[1]. The protocol that we are developing is for Apache Cassandra, is called Accord[2], and forms the basis of our new distributed transaction feature [3]. I will note that the whitepaper linked in [3] is a bit out of date, and there was a bug in the protocol specification at that time. We hope to publish an updated paper in a proper venue in the near future.

[1] https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf [2] https://github.com/apache/cassandra-accord [3] https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15...

Do you anywhere elaborate what you mean by leaderless, and how this affects the semantics and guarantees you offer?

So far as I understand both Kafka and Pulsar use (leader-based) consensus protocols to deliver some of their features and guarantees, so to match these you must either have developed a leaderless consensus protocol, or modify the guarantees you offer, or else have a leader-based consensus protocol you utilise still?

From one of your other answers, you mention you rely on Apache Bookkeeper, which appears to be leader-based?

I ask because I am aware of only one industry leaderless consensus protocol under development (and I am working on it), and it is always fun to hear about related work.

> If you really want to get silly, x2iedn.32xl is 128 vCPU / 4 TiB RAM, and you get 3.8 TiB of local NVMe

This doesn't affect availability - except insofar as unavailability might be caused by insufficient capacity, which is not the typical definition.

> Depending on your availability SLOs, of course

Yes, exactly. Which is the point the GP was making. You generally make the trade-off in question not for performance, but because you have SLOs demanding higher availability. If you do not have these SLOs, then of course you don't want to make that trade-off.

I think you may have misunderstood the GP and are perhaps misusing terminology. You cannot meaningfully scale vertically to improve write availability, and if you care about availability a single machine (and often a primary/secondary setup) is insufficient.

Even if you only care about scaling reads, eventually the 1:N write:read replica ratio will become too costly to maintain, and long before you reach that point you likely sacrifice real-time isolation guarantees to maintain your write availability and throughput.

This doesn’t seem to provide higher write availability, and if the read replicas are consistent with the write replica this design must surely degrade write availability as it improves read availability, since the write replica must update all the read replicas.

This also doesn’t appear to describe a higher durability design at all by normal definitions (in the context of databases at least) if it’s async…?

Fair enough!

Ha, as it happens there's documentary evidence online from Diego himself, that he was not influenced by VSR.

https://groups.google.com/g/raft-dev/c/cBNLTZT2q8o

A good example to illustrate this perhaps is Babbage. He invented the computer first, but nobody using computers today was influenced by him, impressive though his achievements were! Nor would we say that computers are a kind of Babbage “analytical engine”. We say they are a kind of computer.

I don’t interpret those words that way. I see that as a recognition of the VSR paper, as had been recently highlighted in VSR revisited at the time of publication. I guess you would have to ask the author if VSR had actually influenced his work, it’s certainly possible, but not the inference I would make from that snippet.

The paper references Paxos something like 100 times, versus 3 for VSR. It defines itself as a more understandable alternative to Paxos, so it was certainly influenced both by the existence and relevance of Paxos, and also in opposition to its apparent difficulty.

I know it was published first, we’ve talked about this before :)

But, I’m not sure what was published first decide what’s a variant of what. I would say that, given the breadth of research into variants of Paxos and the ways it can be modified, it is most meaningful today to say they’re all variants of Paxos.

VSR having had little to no research or industry application until recently has a pretty weak claim. It does not appear to have influenced either Paxos or Raft. Raft was influenced by Paxos, and even VSR revisited discusses it in relation to these protocols.

TigerBeetle uses VSR, which is basically a variant of MultiPaxos/Raft.

I’m not sure how a written constitution that is anyway interpreted by “the prevailing political elite class” is functionally much different?

I think the problem is that exploit can mean multiple things, and it’s obviously true that companies want to exploit everything in the non-pejorative sense.

The problem is transforming the word’s meaning in the next sentence to imply they use the resource/personnel unfairly, which is demonstrably untrue as you point out (though it’s certainly the case that for some companies both meanings apply)

Yep, if the work itself is what drives you rather than the organisation and people, and you’re able to achieve that solo, then the question is just whether you prefer a commute and office experience. Personally, while I miss colleague interactions, I find offices and commutes truly depressing, so I would never choose that over remote all else equal.

We do currently require all threads to be created by one of our own factories, but that's primarily because this grew out of a non-byte weaving approach (where we explicitly replaced our concurrency primitives). Looking at the class now, all of its state could easily be stashed in either global or ThreadLocal variables, so I don't see anything that would stop us working with FJP etc.

> Additionally, modifying application code just track thread creation isn’t ideal.

This would certainly be necessary, but don't you anyway need to rewrite the application to trap synchronised, volatile, atomic accesses etc? It doesn't seem all that different to rewrite calls to Thread::start. The issue of JVM threads is perhaps a little trickier, but I am not averse to some ugly integrations. Just take a look at how we make RNGs deterministic

> So it is worth to give them a try.

Thanks for the tips! I am not sure when I will have time to apply these techniques to our simulator, but they are no doubt valuable for the protocol simulations I am relying on today, so maybe I will have a justification to explore them sometime soon.

Really cool work too. I hope it manages to make its way into more hands, so that this technique can be used more widely.

We have something very similar[1] we use in the Apache Cassandra project to test complex cluster behaviours.

We appear to use exactly the same basic technique, using byte weaving to intercept concurrency primitives such as synchronized, LockSupport etc to pause the system thread and run them on some schedule.

We only currently run (deterministic) probabilistic traces though, we can’t search the interleaving space. But the traces for a whole cluster are extremely complex and probably unsearchable.

I have been meaning to publish it for broader consumption for years now, but there’s always something more important to do. It’s great to see some dedicated efforts in this space.

[1] https://github.com/apache/cassandra/tree/trunk/test/simulato...

I have always assumed it was just due to the uneven distribution of weight by vehicles often being stationary in the same spot, so those points are subject to more compression forces than the surrounding road surface.

If it were acceleration and deceleration I’d have expected the effect to be less localised, as breaking and accelerating happens over a much longer distance.

But, I have no actual idea. It’s just probably not friction…

This assumes that road wear and tyre wear are caused by the same mechanism, but tyre wear is presumably caused predominantly by friction, whereas road wear is at least in part (and perhaps predominantly) caused by compression, creating pot holes by causing the earth underneath to move.

Where does 110PW come from? Wikipedia quotes global energy production at 19.6TW which makes 3.7TW in five years not too bad?

There are always local subsidiaries, so these are generally European companies making the profit and paying no tax. Several of these companies famously exploit jurisdictional loop holes with how they define where the value arises to avoid paying much tax anywhere. While there are rules around how they can do this, they do seem to be broken.

I’m simply providing an alternative view; this is about perception of relative risks, so there is not an absolutely correct position. It might also be that in Tokyo the speed of traffic is higher or road users less conscious of cyclists so the risk of collision is worse, or that the conformance of pedestrians to the shared path rules is better, but in London my experience is that cyclists aren’t that fond of the shared paths.

Most collisions with cars are also not even likely to be as dangerous as you suggest - the really hazardous traffic is lorries, buses etc. I think you also downplay the risk of serious injury when colliding with a pedestrian - to both of you. This also confers potential legal risk, as you are the road user likely to be liable in this jurisdiction should a vulnerable path user incur a serious injury.

No doubt risk averse cyclists like mothers with children are likely to prefer these shared paths whatever the prevailing conditions, but that doesn’t mean they are outright superior and the specific context is likely to matter a great deal.

Bicycle paths on walkways are also hazardous to cyclists - and stressful, at least in London where there’s a lot of pedestrians and they uniformly ignore the markings and treat them like any other path. Crashing into a pedestrian is certainly less bad than being crashed into by a car, but most cars do pay attention and try not to hit you (regrettably not enough, admittedly), and you can make good progress.

I think I prefer to be on regular unseparated road, at least in busy areas where road users are used to cyclists. Rural or suburban areas are a different matter, the relative speeds of road traffic and number of pedestrians no doubt flips the trade off.

Are you thinking of Hinkley Point C? My understanding was that there has been no additional funding provided. The project is way over budget, but it is EDF and its owner (the French taxpayer) who appear to be on the hook.

The project is funded by a pre agreed electricity price for the plant once it is operational, and private investment.

It is however likely to impact negotiations for Sizewell C which has not yet been greenlit, as funding terms amenable to the U.K. government and EDF have not been reached. EDF want terms that would cause cost overruns to be born by energy consumers, to avoid a repeat of Hinkley Point.