Fast-forward 4 months, the project had grown and scaled to support the influx of new curious devs and use-cases that stretched the bounds of what was possible at the time. At the end of the 4-months, we had a 128 node cluster that stayed up for years and still powers all of the emergency notifications in a few states in the US!

https://github.com/etcd-io/etcd/blob/master/Documentation/pr...

(The embedding mechanism is copy-paste, which I find both ingenious and a bit distasteful. Maybe I’m just sore I didn’t think of it first)

We spent quite some time evaluating different raft and paxos implementations (mostly Consul and etcd raft libraries), and found out etcd to be the most stable and flexible for our use case. It was very easy for example to swap the transport layer to use grpc. The fact that etcd implementation is represented as a simple state machine makes it also much easier to reason about under complex scenarios for debugging purposes, instead of digging into multiple layers of abstractions.

In retrospect, this came with quite a learning curve. We've had to deal with issues caused by our own misunderstandings on how to use the library properly. At the same time the fact that the developers favored stability as opposed to user friendliness was exactly what we found attractive using etcd's raft. Additionally, CoreOS developers were super friendly and helpful to help us fix these issues. We've reported and fixed some bugs as well. Kudos to them for all the help they provided at the time.

[1] https://github.com/abronan/proton [2] https://github.com/docker/swarmkit/commit/89de50f2092dfd2170...

What I remember is, during DockerCon in June 2016, I went into the code to see how it worked, and I found a top-level file setting up data structures and handlers that seemed to be 90% the same as the equivalent file in etcd. And the underlying implementation is reused via vendoring.

Maybe this rings a bell with you and you can tell me what I saw, because I can't find it now.

Maybe I dreamed the whole thing.

I did, and still do, think integrating etcd into Swarm Mode was a masterstroke; we had spent the previous two years working to avoid "first you must install etcd" in a different way that nobody got. Afterwards we created kubeadm to ape the 'init' and 'join' functionality.

Any thoughts on rkt?

The Container Networking Interface was directly created by the work in rkt and continues on today inside of Kubernetes and the CNCF. This work made it possible for an ecosystem of networking solutions to exist that could take advantage of everything Linux has to offer.

The creation of the Kubernetes Container Runtime Interface (CRI) was also spawned, in part, by the existence of rkt and the need to consider container runtimes for use with Kubernetes. It was a long hard engineering effort but I think the separation that CRI forced the kubelet to go through and the competition of various runtimes is good for the ecosystem and the resilience of the Kubernetes project.

It is very unlikely that rkt will be part of the Kubernetes ecosystem at this point with the existence of containerd, and CRI-O as Kube CRI solutions on Linux. And there were missed opportunities on a variety of fronts along the way. But, rkt continues to be used by many organizations for other niche use cases of containers. And the shifts that rkt caused above were positive improvements for the Kubernetes ecosystem.