To some extent, the distributed consensus is kind of the important part. If you have a bunch of components that don't know what state they're supposed to be in, you don't really have a cluster. Looking at the marketing documentation (I didn't read the code), they just wrote their own consensus and datastore instead of using etcd. So the underlying fundamental computer science problems still exist, but nobody has been burned by this particular implementation yet. That doesn't inspire much confidence for me; if etcd blows up, at least someone else's cluster has blown up in the same way before. Now if it happens to you, you get to be the first person to debug it. Fun.

In general, I'm not blown away with Canonical's track record. I used microk8s in its early days and it kind of blew up my coworker's computer. Networking stopped working. Making a request to localhost:5000 would return an nginx error page, even though nginx wasn't running anywhere on that machine or the network. It even persisted itself through reboots! We just reinstalled the machine eventually. It was weird stuff and I haven't touched it again. (I prefer kind locally and k3s for small "real" clusters.) Then there's Snap, major breaking changes to Ubuntu for no reason, etc. I just don't trust Canonical much, and I'm not quite ready to take a leap of faith on their new distributed database. But maybe it's great, and we'll all be using this in a year. A stopped clock is right twice a day.

> I just don't trust Canonical much...

They lost me when I've tried their ubuntu server when I was lazy one day and greeted with their Landscape advertisement in the MOTD display.

Then I installed armbian's ubuntu version because Debian version was not ready and found out that MOTD was downloaded from web every time I log in.

Add analytics (now opt-in), forcing snaps and their silent-ish efforts to monopolize the landscape, I avoid them altogether.

I feel an urge to note that ubuntu server is a decent operating system.

We use it everywhere in my current place of employment and are very satisfied with it.

(maybe it's because we deploy servers using pxe — it makes good ubuntu installs, with no ads in motd & no snapd)

Ubuntu server is decent but, Debian is better IMHO. I'm using stable for servers and testing for desktops for 15+ years (eh, wow. time flies).

Deploying servers with PXE is fun though. We use PXE and XCAT to manage our fleet. Commissioning ~200 servers in 15 minutes with three commands while sipping coffee is really satisfying.

I work in this space (being purposefully vague), you are getting a customized image of some sort - PXE really isn't related in this way, that's just a method to get the image onto the destination disk. The official Canonical Ubuntu Cloud Images produced for use this way have the MOTD behaviour and snapd in place based on my exposure to them. https://cloud-images.ubuntu.com/

we are using this thing — http://archive.ubuntu.com/ubuntu/ubuntu/dists/focal/main/ins... , it is loaded with PXE.

It doesn't put a "customized image of some sort" on servers, it literally iterates through installer steps with the autoanswers provided by preseed file.

You might say that it is customized since we wrote a preseed file. Well maybe, but it's not a "customized image". Also we didn't write any specific code to remove unwanted packages. It just doesn't install them. No weird packages like byobu :)

nod I'll go out on a limb and say "I think you may be somewhat in a minority in 2020" as many in my travels have moved over to an image based deployment methodology - it's not absolute and of course I've not been in every company, just the landscape in general. I wasn't aware Canonical had an "online PXE" flavour that you linked, TIL.

I say all this above, but I'm sure there are still shops out there burning ISOs to CDs (maybe USB now!) and hand installing everything - if I've learned anything, it's that nobody seems to agree on how to do stuff the same way. :) Each company is it's own beautiful unique snowflake full of their own design and deployment patterns.

You may be in for a shock using Ubuntu Server 20.04 and later as it has switched to using cloud-init. Preseed files no longer work.

You should be able to use the "Legacy" (their new words) ISO installer. I'd be interested in your results if you get a chance to followup here. http://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.0...

The new format does seem much nicer to my eyes but I suppose tastes vary. A pain to have to change but it looks like good work and a new release is usually required some pressed Hackett anyway.

They do work. Old installer is marked as legacy but still works.

Same story with IBM OpenShift and their telemetry.

As the person who has worked on making sure telemetry is useful and valuable for users and customers (helping drive insight into quality of kube and close the loop on fixing persistent issues), can you provide some more details about how we’ve let you down? We tried to be as responsible as possible but obviously we’ve failed along the way - what can be done to improve it?

Make the telemetry opt-in to begin with.

The comment I am replying to is complaining about Ubuntu calling home, IBM OpenShift is the same story.

that's what they call 'opinionated default configuration'; then they tie in subscription management with telemetry so that opting out of it isn't an option to begin with; https://docs.openshift.com/container-platform/4.1/telemetry/... Several further tie ins and the system as a whole becomes barely usable. But then integration is the whole point of openshift, isn't it?

Wow, I’d didn’t realize the 4.1 docs didn’t get updated - newer doc versions are correct in that disconnected subscriptions just manually entered via OCM. 4.1 didn’t include the support for disconnected clusters and at the time the docs were correct. Newer versions are more clear.

https://docs.openshift.com/container-platform/4.5/support/re...

I will follow up on opt-out being the default for the evaluation version of OpenShift. It is already opt-in for OKD.

Not to be snarky, but that's easy. Just disconnect your cluster from the internet. We have hundreds of customers that are running disconnected clusters.

And that applies to Ubuntu just as well. All I'm saying IBM is no better than Canonical in this regard.

Not complaining - If you want me to complain about IBM cluster technology that would be a whole different story.

And no I'm not gonna ask about disconnected clusters :).

Haha. I still work for Red Hat, and Consulting at that. My pay is determined by how many billable hours I put in and my customer reviews. Not directly by how much you buy.

I thought there is no Red Hat anymore.

Red Hat is a subsidiary of IBM, not acquired and absorbed. It is very much its own thing within the company as a whole.

I really thought it was a merger. Is Red Hat still a seperate legal entity?

Yes, separate legal entity, combined financial entity (although I know little about the financial details).

Putting a single commercial, once, on the MOTD file is a sure way to tarnish the reputation of a whole distribution and its derivatives forever.

The worst thing about Linux are its users.

Care to elaborate?

It's more of a last straw than a sole reason to be honest. There's a lot of stuff Canonical is doing which can be considered as embrace, extend and extinguish.

It's funny when a corporation uses tactics of another corporation it wants to beat.

[0]: https://bugs.launchpad.net/ubuntu/+bug/1

It’s easy enough to configure motd to pull your own notices for a company, and useful too. Nothing there seems particularly weird to me in this day and age, and Ubuntu has a much better track record of actual security maintenance over the long term.

> Ubuntu has a much better track record of actual security maintenance over the long term.

Debian has 24 hour security fixes for 15+ years. It also supports "OldStable" in terms of security & backports. For some time it also has "Long Term Support" teams which supports older releases.

Debian is "the original" install & forget distro. Ubuntu has some commercial sauce over it but, unless you have a special need, Debian can handle everything you throw at it.

We have so much servers so that we sometimes forget some of our background service servers' and they hum all-along with all security updates applied.

Microk8s uses dqlite as a distributed database, which uses RAFT for consensus. They have their own implementation called C-raft. So if their implementation is decent it should be fairly comparable to etcd’s raft implementation.

If etcd blows up, your cluster should stay in the same state it was in before etcd blew up.

Presuming, of course, that your cluster is completely isolated from the outside world. Otherwise, the interaction of various controllers, deployment stacks, and application usage conspire to generate endless amounts of cluster state change requests, which can quickly pileup if etcd is unavailable or behaving erratically and cause all manner of havoc. Containers can fail to restart after crashing; jobs won't run, etc. Plus, in k8s etcd is typically used for more than just the barebones state required for basic node and pod status. Metrics, API requests, DNS, etc, often depend on etcd--the same etcd--for storage and communication, which can compound the effect of a flakey etcd and even instigate it.

> In general, I'm not blown away with Canonical's track record.

Yeah, they really wanna be like Apple, but end up being more like Google, starting a lot of projects and abandoning them in couple of years.

You don't need distributed consensus to have a cluster of reliable services. We've had large-scale clusters of applications for what, 30 years? And distributed consensus being generally used by a few players for about 10 of those. Etcd is neat, but in no way mandatory for what it was even invented for. 99% of people just don't need it.

I don't know of any clustering system that works without some sort of leader. In many cases, the leader is you. If your computer dies, you can walk to another one and still deploy software. If you disappear, that's a Big Problem, however.

No modern clustering system is much different. Services aren't killed off if a leader can't be elected. It's just that having all your servers alive but uncontrollable is almost as scary as them being down.

Distributed consensus doesn't require a leader. The same process used to elect a leader (i.e. change leader state) can simply be used to change state generally. See, e.g., There Is More Consensus in Egalitarian Parliaments: https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf

In fact, for something like what etcd is used for in k8s, as the root and lynchpin of all state, but not direct application I/O (pods use their own data stores, and even for cluster management etcd usually just contains pointers to external data), a leaderless architecture makes the most sense. Protocols to choose a leader are latency and throughput optimizations, but the core state for a cluster (members, pod metadata, etc) shouldn't require very much state and therefore should be relatively low traffic as compared to most database applications. And as shown in the above paper, for certain scenarios (including, arguably, the k8s scenario) a leaderless architecture can have better latency and throughput, not to mention availability.

If you run etcd as a singleton you already get this behavior - consensus is shortcircuited and you only have to pay the cost to write to durable storage (which is already heavily batched). And you need durable writes so you can crash recover (kube has a gaping hole today in that a restore from an earlier time point breaks many controllers until a reconcile is performed).

Note that Kubernetes requires a total ordering of writes (which simplifies how hard it is for us puny humans to reason about) AND requires strong consistency in order to provide guarantees like “this pod only runs on one machine at a time” and “PVs aren’t released until the pods are really stopped”. Leaderless is a simple tradeoff - singleton or three instances. That’s the best possible choice in the world and it’s etcd and it’s relative simplicity that make it possible.

I’ve never seen a production Kube system with HA etcd go down due to non-human error, so I don’t believe single instance is going to give you better availability when single machine faults happen (they happen frequently; about 0.5-1% of the machines in the OpenShift fleet - cloud and on-premise - are down at any one time due to power, software, or hardware issues). Almost all of those clusters tick along fine when they lose that machine.

> I’ve never seen a production Kube system with HA etcd go down due to non-human error

I have, with OpenShift, several times. The worst one took hours to get it partially back online, days to fully recover. At least I got to have one free drink before I had to leave our holiday party to start fixing it. (Ok, technically it was non-prod; luckily prod was ECS... long story)

> I don’t believe single instance is going to give you better availability when single machine faults happen

Depends on your definition and context of availability. It is possibly the simplest thing to destroy any existing nodes (and in the event that a node can't be destroyed, null-route it and/or disable its network port) and bring up a new node. So single instance is fairly easy to recover. But with multiple nodes, for each function of each instance that is required to reach consensus, the likelihood of consensus failure increases; you actually need more nodes and variety in the cluster to resist consensus failure. Yet ironically, the more nodes you have, the higher the probability of failure. In the end, the real-world reliability of the system is based on additional factors besides the network model.

The database in question, dqlite, is raft-around-sqlite, and that’s just about the most widely used sql database anywhere, albeit pretty thin. Recovering files on disk won’t be an issue with dqlite. It would be interesting to see a jepsen analysis of dqlite to assess resilience in the face of trouble.

fwiw I think k3s also has an option to use canonical's dqlite.

Edit: they deprecated the option https://rancher.com/docs/k3s/latest/en/installation/ha-embed...

Bummer. Would love to know why though.

A long wished for feature, but closed wontfix a long time ago: https://github.com/kubernetes/kubernetes/issues/1957 :-(

That's a pretty soft wontfix though. Might be reopenable at this point.

We talked about it recently, and certainly people want to do this, but I don’t see it happening anytime soon in core. Downstreams are free to carry patches though and many do.

It’s more that the core team doesn’t have the time to support these, and we already have a fairly tight contract with storage, and we still find edge cases that need to be fixed, so the extra cost for the committers to support this is high.

For single node sure, but the moment you have some set of jobs requiring a multi node setup, the chance of wanting to find yourself in a situation where the cluster has essentially bricked itself due to a momentary power outage reduces to zero really quickly.

The only place etcd might not pay off is in disposable dev environments or something, but do you really want your prod setup to page only to discover a complete cluster rebuild is necessary to resolve the problem?

There are various levels of outages you may encounter with an etcd outage, but it shouldn't take your workloads completely offline. The cluster API will be offline, but the workloads will keep on chugging. I think with the decoupling of etcd and the expectation that this might happen, we'd see more improvements on how to (gracefully) handle these situations.

Also, there was a bug with kube api where it wouldn't failover to other etcd members during an outage. So I would say most customers have ran kubernetes with a "single backend" at some point.

If K8S can be made turn-key, it will become possible for some deployments to realize massive cost savings by deploying across clusters of bare metal machines from hosts like OVH, packet.net, datapacket.com, and so on. The cost of bandwidth and processing power is just so much lower.

The only remaining headache would be database. Setting up reliable mission-critical HA Postgres is kind of a pain, and it's nice to not have to worry about it. If that can also be canned, then this kind of rig would be a really compelling alternative to the managed clouds... unless you need other things like S3, etc.

The one thing I wish upstream K8s would implement is embedded etcd so that the operational complexity becomes a bit more manageable. Luckily k3s has that :)

K8s does have a well-defined storage abstraction that wraps the operations it performs.

Surprisingly, Etcd is neither scalable or high performance in our small cluster with <10 nodes on GKE. It had been the single most frequent source of prod outages.

But it appears that k8s community do not section a list of compatible storage engine that can plug and play with k8s. Or I might have missed some recent development.

k3s recently (v1.19) deprecated built-in dqlite support. (it was replaced with built-in etcd.)

though with kine (kine is not etcd) you can easily use mysql/sqlite/dqlite/postgresql.

https://github.com/rancher/kine provides a translation layer for etcd that supports various SQL flavors. It is currently used only by k3s afaik