In my opinion, the opposite is true about Kubernetes (though it may be true about the Mesos stack). It's a great way to reduce the need for dedicated ops engineers.

Once you have a Kubernetes cluster running, it's essentially a "platform as a service" for developers. They can deploy new pods and configure resources all with a single client. You generally no longer need to think about the servers themselves. You only need to think in terms of containers. No package management, no configuration management (in the Salt/Puppet/Ansible/Chef sense), no Unix shell access needed.

It's hugely liberating to treat your cluster as a bunch of containers on top of a virtualized scheduling substrate.

Kubernetes itself is very lightweight, and requires minimal administration. (If you're on GKE, K8s is managed for you.)

Swarm Engine is very much knocking off Kubernetes' features. Swarm offers the convenience of building some orchestration primitives into Docker itself, whereas Kubernetes is layered on top of Docker. Other than that, they're trying to solve the same thing. I'd say Kubernetes' design is superior, however.

There aren't a lot of "Here's an API to throw up your containers and provision IPs" services out there. I see that Amazon has one (I avoid Amazon; they're the new Wal-Mart), but no one else really. I mean DigitalOcean provides CoreOS, so in theory you can provision one of those droplets and toss up some containers, but there's not a real API for "use this endpoint to deploy your container."

In the corporate world, yes .. we have dev ops teams to try out and create provisioning systems to throw up Mesos or CoreOS or OpenStack clusters. Once they're up and your devs are using them, they're lower maintenance that provisioning individual servers for individual services. For home devs, it'd be nice to have a plug an forget for docker containers (other than Amazon).

The other thing I still don't get about Docker: storage. MySQL containers have instructions for mounting volumes, but it doesn't feel like there's a good, standard way to handle storage. I'm sure that makes it more versatile, but like I said previously: plug and forget for home devs and startups. You want a databases as a service, get ready to shell out $80/month minimum for Amazon RDS or create a set of scripts around your docker containers that ensure you have slaves and/or backups.

> a real API for "use this endpoint to deploy your container."

I've been using Triton since as early as possible and it's fantastic.

You can tag/label containers and they show up in the SDN DNS by tag (called CNS), no need for Consul or other service discovery.

I do still use Consul for many reasons but CNS is awesome for bootstrapping a cluster as well as public dns routing/failover.

Also true about Docker Swarm.

> There aren't a lot of "Here's an API to throw up your containers and provision IPs" services out there.

I strongly believe this (in particular, "hosted Kubernetes as a service") will start arriving soon. It's a great business opportunity.

> The other thing I still don't get about Docker: storage. [...]

Kubernetes solves this. For example, if you're on GCE or AWS, you can just declare that your container needs a certain amount of disk capacity, and it will automatically provision and mount a persistent block storage for you. Kubernetes handles the mounting/unmounting automatically. Kill a pod and it might arise on some other node, with the right volume mounted.

Either run a seperate vps for MySQL or use a hosted solution.

Containerization will get there but i think putting old some types of old school apps in containers is a bad idea.

E.g. redis, beanstalk, and similar fit nicely I to clusters and restarting nodes without too much issue or downtime.

MySQL, if your install is large enough, you never want it to go down. This pretty much goes against everything to do with containerization, your developers also gain no benefit of it being in a container (apart from maybe running a small db in developent only where Provisioning a vm would be too painful)

Maybe that's the difference between a product and a solution. I don't know.

Kubernetes admittedly takes a slightly different, modular, layered approach, whereas Swarm is simple to the point of naivety.

This simplicity is potentially a threat to its future ability to adapt to different requirements, whereas Kubernetes offers a separation of primitives that allows it to scale from "simple" to "complex".

For example, in Kubernetes, a deployment is a separate object from the pods themselves. You create a deployment, which in turn creates a replica set, which in turn creates or tears down pods as needed.

But you don't really need to work directly with replica sets or know exactly how they work, but they're there, and can be used outside of a deployment. If all you care about is deploying apps, then you only need to deal with the deployment.

Exactly. To me, that approach is what allowed Docker to make containers on the laptop ubiquitous. Kubernetes is unlikely to be the tool that takes schedulers to that universe. Swarm might and I think that's the goal.

I mean, I don't really want to care about pods and replica sets. They're obstacles in between me and what I want when what I want is more horsepower behind an app. For the same reasons I probably am better off with garbage collection than malloc [1]. I've only got so many brain cells.

[1]: http://linux.die.net/man/3/malloc

I'm sure Swarm has its place, though.

(1) Write your own custom deployer https://github.com/matsuri-rb/matsuri

(2) Write your own controllers https://deis.com/blog/2016/scheduling-your-kubernetes-pods-w...

Those two are just examples. My point is that if you need to dig in, you can. You can do this because those higher-level behaviors are built on a solid foundation of well-thought-out primitives.

What I expect to happen is that we will have a diverse set of things built on top of Kubernetes. Some of it will get folded into Kubernetes core. Some will not. I think the center of gravity for containers has been shifting towards Kubernetes.

My experience of K8s so far is that GKE is happy path and most of the demonstrations/documentation is focused on that use case. When you step off that path, you can either go for something scripted which does a lot of things in the background, or what seems like quite an involved manual setup (etcd, controller node setup, Certificate setup, networking etc)

We _absolutely_ want to push Kubernetes to be "gmail-easy" - here's a recent demo:

https://www.youtube.com/watch?v=Bv3JmHKlA0I

If you don't feel like watching a video, here's the commands to get a fully production-ready cluster going:

  master# apt-get install -y kubelet
  master# kubeadm init master
  Initializing kubernetes master...  [done]
  Cluster token: 73R2SIPM739TNZOA
  Run the following command on machines you want to become nodes:
    kubeadm join node --token=73R2SIPM739TNZOA <master-ip> 

  node# apt-get install -y kubelet
  node# kubeadm join node --token=73R2SIPM739TNZOA <master-ip>
  Initializing kubernetes node...    [done]
  Bootstrapping certificates...      [done]
  Joined node to cluster, see 'kubectl get nodes' on master.

If you have thoughts on how we should do it better, PLEASE join us! https://github.com/kubernetes/community/tree/master/sig-clus...

Obviously there's Kelsey Hightower's great Kubernetes the hard way tutorial, but even there some details are quite GKE specific.

It'd be nice to see some tutorials that make no assumptions about running on a specific cloud provider or using automated tools to bootstrap the cluster.

My suggestion is to design an interface for Kubernetes in ways less reflective of Conway's Law. That's how the needle moves from 'solution' toward 'product'. In essence, that's what I think 'Gmail Easy' boils down to. Gmail has user stories that are based on people (much)+ less technically sophisticated than a new fresh from school Google SRE.

I think that the call to action suggests why Docker is pursuing Swarm.

In a decade of container orchestration development, creating a good onboarding experience with the diverse demographic of Docker users has never been a priority within Google. If it had been a priority, Gmail-easy Kubernetes integration might have long since been a pull request from Google. That's what it says right on the label of the open-source software can.

The talk about forking Docker seems consistent with the absence of pull requests. It's just another business decision.

Deployment of a container to a node is roughly an atomic transaction. If it fails due to a network partition or server crash etc. The container can just be deployed again to the node again. By comparison, a partially executed provisioning script can leave the target node in an arbitrary state and what needs to happen next to bring the node online depends on when and how the deployment script failed as well as the nature of any partial deployment...and whatever state the server was in prior to deployment.

But if you use your production deployment scripts for development deployments, then that problem has been dealt with one way or another.

We're still hesitant to deploy docker in production though, but its on our list. For now we rebuild the production environment from the recipes in the Dockerfile and docker-compose.yml.

Before that we were using VMs with vagrant. I never really liked how ressource intensive and slow they where. And without VM I also don't see the need for Vagrant anymore if I can use docker directly.

So after 2 years of using docker I'm really very happy with our development setup.

It's just adding yet another layer to the stack, so another interface to learn. And if things don't work you have to deal with the “low level“ docker setup anyway.

Seriously, I've never looked back at Vagrant ever since.

But, that goes against the main hype driven development tenant. If it is on the front pages of HN, CTOs should quickly scoop up the goodies and force their team to rewrite their stack without understanding how technology works. Things will break, but hey, they'd be able to brag to everyone at every single meetup how they are using <latestthing>. Everyone will be impressed, and it also looks good on the resume.

/s (but I am only half-joking, this is what usually happens).

I also like using multiple vendors for a tool stack. Helps keep concerns separated.

For internal apps, that might be way better and cheaper than to run on AWS or GCP.

On the other hand, I don't think that roll-my-own shell scripts are going to solve the problem Swarm/Kubernetes/Mesos address -- scheduling -- as well as those tools will. The reason I think that is that many scheduling problems are NP-complete [1] and dynamically optimizing a scheduler for a particular workload is a non-trivial algorithmic challenge even at the CS Phd level.

[1]: Wikipedia lists five 'scheduling' problems: https://en.wikipedia.org/wiki/List_of_NP-complete_problems

I disagree. I worked with Kubernetes on a small team, and its core abstractions are brilliant. Brilliant enough that Docker Swarm is more or less copying the ideas from Kubernetes. Even so, Docker Swarm just isn't there yet.

> As I see it, Swarm seeks to solve orchestration analogously to the way Docker seeks to solve containers.

That's how I see it too. However, multi-node orchestration is MUCH harder. The Kubernetes folks know that. Setting up K8S from scratch on a multi-node setup is still difficult, and the community acknowledges it. Contrast that with the way Docker is handling it. Docker seems to be trying to sweep difficult issues under the rug. Look at the osxfs and host container threads on the Docker forum as examples. People had to ask the developers for more transparency before it started showing up.

I don't blame Docker for how they got to "sweep difficult issues under the rug", since that is part of their product design DNA. It's just that things are coming apart at the seams.

> Google doesn't see a business case for pushing Kubernetes toward the Gmail end of the ease of use spectrum.

That's correct. That is what GKE is for. On the other hand, that is what Deis/Helm is for. Turning the project over to the Cloud Native Foundation, listening to the developer community ... there is a lot of things Docker can learn from the Kubernetes project when it comes to running an open-source project.

  They're [Kubernetes, Mesos, etc] not so great for a small team [or individuals] who are just trying to deploy some software