That sounds like something Al Vermeulen would say. Though my recollection of the quote is more along the lines of:
"There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures."
I'm guessing the talk it came from is Under The Covers: Core Distributed Systems Primitives that Power Our Platform[0]. That presentation is full of great insight into how AWS thinks about distributed systems. The other presenter, Swami, is the VP of AWS Databases now but started off getting Amazon's initial Paxos implementation going. I was lucky enough to work on those core building blocks and it was an incredible experience.
That work must have been really gratifying especially since it underpins so much now.
> I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.
As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer).
One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO.
If you do build a plugin system though, you can pass off the work of maintaining the other implementations!
[0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all
“There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.”
Obviously a bit of an over simplification since there have been some advancements in that area but not far off.
This article introduces an approach that falls under the second mode and I think it’s a good way to build a distributed system without having to re invent the wheel especially if you can just rely on the data store for the consensus and make your brokers and nodes relatively dumb.
Look at k8s and etcd for a very widely used example of this approach as well.
I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.