News for hackers. Stuff that matters.
Re:Why we rolled our own consensus algorithm by eloff on Thursday November 15, 2018 @01:00PM (#18461570)
This seems like a really, really bad idea.

However, if everyone had that attitude we'd never improve on the existing art. So I'm glad some people are willing to risk it.

But just to reiterate, if you're thinking of rolling your own consensus - or even implementing an existing consensus algorithm yourself (although that's much worse for Paxos than for Raft) - you're almost certainly making a big mistake.

Re:Why we rolled our own consensus algorithm by zzzcpan on Thursday November 15, 2018 @01:36PM (#18461923)
If you have enough background in distributed systems to choose a consensus algorithm you are definitely capable of implementing one yourself and even invent your own. It's not a bad idea either, most existing consensus-based systems are absolute garbage. People just hugely overestimate how much those algorithms matter for production systems.

EDIT: Note, I'm not talking about blockchains and byzantine fault tolerance, but in context of Paxos/Raft.

Re:Why we rolled our own consensus algorithm by eloff on Thursday November 15, 2018 @01:46PM (#18462030)
That's like saying if you have some crypto background you can implement your own crypto library. It's TERRIBLE advice.

Read the Google paper on their chubby distributed locking system using paxos. They had a hard time getting it right - unless you think you're so much better than them. IIRC Leslie Lamport works (worked?) for them.

Use an existing, well used, well scrutinized implementation of a known algorithm.

Re:Why we rolled our own consensus algorithm by kjeetgill on Thursday November 15, 2018 @01:52PM (#18462070)
Are you sure? I thought Lamport has been at Microsoft Research since forever.
Re:Why we rolled our own consensus algorithm by therein on Thursday November 15, 2018 @03:08PM (#18462824)
You're correct. Lamport has joined MSR in 2001 right after publishing "Time, Clocks, and the Ordering of Events in a Distributed System" and has been there ever since.

I am not sure if he is still at the MV office but he was one of the reasons why MSR kept a presence in MV for a while.

Re:Why we rolled our own consensus algorithm by lou1306 on Thursday November 15, 2018 @08:03PM (#18465222)
Actually "Time, Clocks..." is a paper from the late 70s. Maybe you were thinking of the Paxos paper, "The Part-Time Parliament" (1998)?
Re:Why we rolled our own consensus algorithm by therein on Thursday November 15, 2018 @08:56PM (#18465606)
Yeah, copy-pasted the wrong title and by the time I noticed, it was too late to edit. :)
Re:Why we rolled our own consensus algorithm by eloff on Thursday November 15, 2018 @04:20PM (#18463462)
Ah the vagaries of the human memory.
Re:Why we rolled our own consensus algorithm by apersona on Thursday November 15, 2018 @04:03PM (#18463287)
If you're a normal software engineer, I agree, you shouldn't implement it yourself.

But if you're a researcher/had a CS research background, I don't see why not. By definition research results in novel approaches (ideally better) than the current ones. You have to be willing to challenge the consensus.

PhD students who can prove that their algs work should work on it, like the results of this paper: https://raft.github.io

Re:Why we rolled our own consensus algorithm by dlubarov on Thursday November 15, 2018 @01:54PM (#18462090)
> Use an existing, well used, well scrutinized implementation of a known algorithm.

Keep in mind that this is BFT consensus. Until recently, it was a niche area of research with very little real world usage. Even if an existing algorithm like PBFT met their needs, there are no well tested PBFT libraries.

Re:Why we rolled our own consensus algorithm by drfuchs on Thursday November 15, 2018 @02:50PM (#18462650)
No, Lamport does not now, and never has, worked at Google.
Re:Why we rolled our own consensus algorithm by cremp on Thursday November 15, 2018 @01:32PM (#18461885)
Just to drive it home.

Those who actually make their own encryption libraries know their stuff, and have been in the space for years.

You should never, ever, roll your own encryption.

If it's a requirement, then change your damn requirements, because just like consensus; it is not easy to get right, and the pros make mistakes.

Re:Why we rolled our own consensus algorithm by zzzcpan on Thursday November 15, 2018 @01:41PM (#18461980)
Consensus is not crypto, it's ridiculously trivial in comparison and usually so useless, that nobody even bothers doing it correctly all the way to the end user.
Re:Why we rolled our own consensus algorithm by sagichmal on Friday November 16, 2018 @07:29PM (#18473077)
Consensus is at least as hard as crypto.
Re:Why we rolled our own consensus algorithm by bradhe on Thursday November 15, 2018 @02:27PM (#18462401)
bingo.
Re:Why we rolled our own consensus algorithm by dlubarov on Thursday November 15, 2018 @08:02PM (#18465220)
I think it's worth distinguishing a few cases here:

1. Reimplementing an existing cryptographic primitive, such as SHA-256.

2. Inventing a new cryptographic primitive, with a reduction to existing cryptographic assumptions, and implementing it. A recent example is SPHINCS-256, which was proven secure in the random oracle model.

3. Postulating a new cryptographic hardness assumption, inventing a new cryptographic primitive based on it, and implementing it. Recent examples are IOTA's Curl hash function and StarkWare's Jarvis cipher.

#1 is risky, but at least the risk can be mitigated by having qualified peers review the implementation. #2 is riskier, but still, it can be mitigated by having qualified peers review the proof. #3 seems far more dangerous, since it involves conjecture.

What the Tupelo team is doing is like #2 -- risky, yes, but not comparable to #3.