I think you're missing my point. We're on HN right, where we often discuss the difference between technical merit and market fit. There's a time and a place for technically correct solutions, which Google excel at, but a successful business needs more than that, it needs to capture a zeitgeist, which Google sometimes fail to do, see Google Plus for example.

Of course the technical lineage of Kubernetes is Borg and parents (which I admit I've never heard of), but I put it to HN that Kubernetes' lineage is not what created the market fit. In fact I think it's disingenuous of the documentary to imply that (in fairness it does give time to Docker), hence why I'm so keen to give my version of events.

Your point about how The 12 Factors apply just as well to non-containerised software is exactly the beauty of them! The 12 Factors don't come from management, they're not an attempt (at least directly) to capture profit from a market. They are somehow fundamental axioms about writing software in this current era, they're maybe even self-evident, universal truths. It turns out that there is a fundamental connection between the ability for a piece of software to be efficiently scaled and for that piece of software to be collaborated on. 2 or more developer machines share essential characteristics with 2 or more scaled containers, which share requirements with 2 or more CI instances ― reproducibility, behaviour guaranteed by tests, ease of downloading, security, configurability through ENV.

You see how we felt like Heroku understood us the humble developer, it made us DevOps, all whilst helping us satisfy the bottom line of managers? Kubernetes is truly an extraordinary piece of software, but it hasn't, nor do I believe it ever will, achieve anything of The 12 Factors' lasting legacy.

It is, in my opinion, at best converged evolution. Maybe because 12 factors are such self-evident truths - though if true, I'd at best credit them as formulation into a listicle of the fundamental behaviour of JCL batch programs on JES2/3 (don't have enough experience with other systems of that class to compare, but I suspect certain commonality).

What my point is that while Heroku might be a very applicable relation to some, it's very explicitly a Platform as a Service, in a way pushing a bit back towards the model that we were escaping from (by invoking Rails name) in mid-2000 to cheap full-OS machines, from the confines "PaaSsy" LAMP.

Kubernetes didn't capture a zeitgeist by introducing any of the features (though it definitely pushed containers up into enterprise world a lot). It was in no way first, or even second, despite coming from home of linux containers. It doesn't even force you to use 12 factors - I should know something about it, having done what many called stupid, as in I did "lift and shift" of over 60 "classical" applications that in no way fit 12 factors onto it.

Now, trodding a bit further away from concrete facts and into more a world of personal opinion...

Kubernetes captured the zeitgeist by working. There was immense desire across all kinds of business, developments, etc. for something that fit nebulous label of "private cloud". Often with local deployments involved. And to do it cheaper (or at least, easier to automate) than existing "heavy" virtualisation stacks. Not everyone could use at all, or just burn money on AWS, many VPS providers were more like classic VMware virtualization clusters in experience. Some had money to burn on Heroku and actually fit running on it (or were lucky to fit in the cheap band of usage).

Thus you had OpenStack and the dwarfs: CloudStack, Eucalyptus, OpenNebula, Ganeti, with more adventurous ones building on Mesos or fully custom.

All tackling the problem of, essentially, providing simpler and faster (in turnaround time) way of handling users access to resources.

The most moneyed and successful (at least from pov of Ops) was OpenStack. And it was a hot mess. Upgrading an openstack cluster was a meme, and not a nice one. Meanwhile containers gain steam, for good or bad, and people start to move from moving from the quite bad "single app VM instances", to containerised approach (12 factors already hit harder due to the single-app VM approach popularised by AWS ASGs coupled with cloud-init).

Into this enters kubernetes. Yes, it might seem hard to setup - but it's no OpenStack, it brings no licensing purchasing mess of VMware, it initially kills one of the more annoying aspects (though it will return) which is the overlay networking. It brings unified API, simple operating model (wish I kept the old 1.0-1.4 presentations on hand that went bottom up!). We can deploy stateless services in somewhat sensible way (and if you know how, they can be stateful too). The saner internal design and API commonality can be arguably compared to the effect of the original "heroku" command that allowed you to just deploy from a git repo - just from the point of Ops team.

By next year, we move from simple ReplicationController to ReplicaSets + Deployments, as well as gaining Ingress beta and first rush of "pluggable" componentry in the form of Ingress Controllers. It's arguably the time where you can start dealing with K8s as simplistic replacement for Heroku - except it's not some closed magic operated by the Wizard of Oz, but something you can deploy on bunch of Raspberry Pi in middle of presentation at a conference after suffering a long trip (we even forgot, iirc, to bring our planned kube-master machine and had to improvise). Early versions of TPRs (priors of CRDs) make me give what were probably some of the first talks about operators (before CoreOS people coined the term) - but I used the same terminology as K8s used (Controllers).

By 2017, the recommended way to deploy openstack starts to be "deploy openstack on top of k8s", and DC/OS starts to visibly lose. K8s makes certain things forced upon you, but you find that it doesn't matter and tends to make sense (Pods having one IP, etc.). Secondary/Tertiary adoption follows, bad memes start about kubernetes (like you only need it for scale, that it's cloud specific, etc. etc.) but it doesn't really matter. Soon, operators start flowing in, thanks to simplicity of the API model (this evolves later to things like Crossplane and even separating kube-apiserver into standalone program, to run without rest of k8s). K8s manifests might feel like JCL, but you don't care because of how bloody more complex the same things used to be before.

Heroku might have provided easy way to deploy constrained application for a developer, but, at least in my experience, k8s became the lever with which I could move a lot of servers with little effort, whether to run large batch jobs, complex meshes of microservices, CI build jobs, or build a heroku-style PaaS. As developers learnt to follo 12 factors, the job became easier.

P.S. Funnily enough, probably because accidents of history kept me on the Ops side, but IME kubernetes adoption always had push from the sysadmin side, not necessarily developer side.

P.P.S. As nice as Heroku is, there's a reason why it's my main example of burning money, especially contrasting how it's billed as "cheaper than paying for an Ops engineer". In the world of global pricing, what works as cheap in Silicon Valley is... not cheap in most of the world.

P.P.P.S This kinda went ranty, I guess I should maybe write a blog post on this perspective...

Very interesting about JCL being an older precedent for the 12 Factors, I've never used it, so I'll have to take a look.

I totally know what you mean about OpenStack! So much potential, but just way too complicated. But I'm not sure it's fair to say that Kubernetes was the first project to Just Work™. I never tried Mesos, but it's always seemed more accessible to me. But I did use Deis and Flynn though, and they were a delight in comparison. Deis of course being the home of Helm, Kubernetes' now de facto packager manager.

Also don't you think that Kubernetes explicitly embraced Docker right from the start as a first-class citizen? OpenStack only ever bolted it on as an after thought. Kubernetes knew what the buzz was and made sure to let everyone know that Docker was front and centre. This is why I think it's better to think of Kubernetes as a spiritual child of Docker (and therefore everything that came before Docker) rather than a technical descendant of Borg.

Although Deis and Flynn were certainly already creaking under the weight of what it truly meant to support the full gamut of cloud features, they were heading in the right direction. Google really did knock it out of the park, which I'm grateful for, but I think they did it more with the sheer weight of money and reputation than any innovative technology. Nobody really knew the devs behind Deis and Flynn, but everyone knows Google's reputation. And so we all just came to warm our hands around Kubernetes' fire, and the rest is history.

JCL and JES2/3 provided a bunch of features (of course, very primitive compared to anything newer) that helped run programs in a world before expectations of "wasting precious disk space" on configuration or state that wasn't critical, thus it tended to operate remarkably similar to running 12 factor batch jobs - for example, you would use Data Definition (origin of Unix dd command - it was a joke on JCL syntax, iirc) to specify open files (with named handles) passed to a program (consider passing that through environment variables or parameters in 12 factor app), telling it to run an otherwise stateless binary, and possibly embedding a short command program (interpreted by it) in JCL stream. A program might be preinstalled on disk, or pulled from tape, your data might be mostly on tapes, etc., and your JCL on a bunch of cards that you submit is, effectively, your declaration of environment for it and the system operators take care it's run properly - but outside of that JCL, there might be no other state involved.

Kubernetes might seem to embrace Docker, but in reality it hid the Docker. In fact, kubernetes explicitly avoided using features from docker outside of certain basics - I'd even say it's part of what I liked the most, as I found Docker to be incredibly clunky once past "local development" stage. As early as 2016 we had alternative runtimes that took in docker images (and sometimes not even them) and ran them in completely different matter. It was just a question of time, when Docker will be eliminated from k8s stack, rather than if. The only real thing that k8s took from Docker and not somewhere else was the idea of container images as packaging format (containers having been widely used at google with LMCTFY and other runtimes before, and LXC predating Docker in wider world with even primitive package system).

Mesos had the issue that, at least in my experience, every bit of the stack was separate. Running "just Mesos" wasn't really an option, and every framework seemed to be building with its own APIs, creating a disconnected mess (mind you, my first actual use of Mesos came after already using k8s for a while). What comes as common bits (without operators) in k8s 1.4, involved IIRC 2-3 different frameworks with disjoint APIs on DC/OS. Quite a different experience, even if you used DC/OS which packaged it altogether for you. Mind you, DC/OS arguably also totally killed community around Mesos in my PoV, but that's a question to someone deeper involved in that ecosystem.

As for OpenStack vs. K8s... I can't even compare the scale difference of running "reasonably operational" k8s cluster vs. "openstack that isn't just devstack on dev's laptop". And when it comes to upgrading it really becomes unfunny.

Also, at least early on, it really didn't seem that K8s would become the half ton gorilla it is now, and while people might have not known devs behind Deis and Flynn, it was in my experience heavily the evangelism of early adopters that pushed it, later bolstered by GKE becoming a thing. Despite that and google's supposed reputation (they really, really don't know how to monetize it, instead burning it - otherwise GCP would be much more of a success commercially than it is, not third in public perception of people I talk with), alternative runtimes continued on and people still believed that DC/OS might be the winner, or that a bunch of other platforms might maybe standardize some bits around docker but no clear winner might appear. Admittedly I haven't heard of Flynn, but I do recall Deis, pre-k8s Rancher, Docker Swarm, CoreOS' Fleet, the unlamented OpenStack thingy, and others being regularly discussed.

The one thing that IMO made k8s truly groundbreaking was the kube-apiserver design, from first principles. Yes, a whole cluster is complex, but it builds from simple component ideas, with the simple patterns of apiserver tying everything together (said "simplicity" is grounded thoroughly in control theory and, I wonder if anyone noticed, symbolic AI designs like Blackboard Systems). This meant that evolving that system, first all internally through combined controller binary, then through TPRs and finally CRDs, was easy. Outwardly (there's some extension if you look close) the base k8s API didn't change since version 1.0, despite huge changes in what you could do with it. Something that I don't think any other alternative took care of, with exception of maybe generalized look at AWS APIs (if you squint to blur the inconsistency and try not to notice gaps hinting at private APIs and differences between what's doable from Console and API).

A bunch of that came explicitly from having experience of much more haphazardly designed Borg APIs and their ossification, and it sometimes showed in discussions on github repo (I recall early requests for CPU pinning being refused by understandably vague but useful explanation of how it led to broken design in borg api and that they would recommend shelving till a good interface gets designed)

Tapes and cards!! OMG that puts JCL into perspective!

Sure Kubernetes' API hid Docker, but its marketing certainly didn't. And I'm not sure it's right to say Docker was eliminated from Kubernetes, it was simply made optional. I think it's hard to argue that Docker isn't still the pioneer, like I mentioned Docker is the OG market fit which all the other runtimes are trying to emulate.

So well-designed APIs, which Mesos doesn't have, I can believe that. But I think it's fair to say Flynn and Deis put a lot of effort into theirs. I forgot about Rancher and Fleet! I don't know about their APIs. For sure Flynn and Deis never got as far as CRDs, but they still achieved those basic concepts we use in Kubernetes today; deployments, storage, config, logs, routes, networks, etc.

I'm just still not convinced that Borg did anything new or innovative that wasn't already in some Open Source project before Kubernetes came along.

The EuroSys paper on Borg is a good place to get the background on Borg, how it influenced Kube, and how it relates to other systems: https://research.google/pubs/pub43438/

Well, the issue is that Borg is not public, and isn't direct ancestor of kubernetes - a bunch of design decisions were informed by the Omega project which was an attempt to build a new Borg (for example, pluggable schedulers, a rather core aspect of k8s).

Marketing hanging onto Docker was a great marketing move, even if all that docker brought was essentially images.

As for basic concepts - I believe that the core issue is that Flynn/Deis/Heroku converge from the opposite starting point (it's arguably a common case for all PaaS).

A PaaS wants to provide those high-level building blocks to the end user, but (especially the case with Heroku and other closed source examples), it hides everything else. In a way it is an offshoot of, if much better designed usually, of the classic LAMP experience, except usually with some extras provided - and these days we get APIs instead of crappy web panel (or cPanel, in fact ^_-).

The implementation details of them aren't for end user consumption, and generally the less they leak the better.

OTOH, we have infrastucture-side systems, which start with "I have X resources and I want to utilise them efficiently". Those tend to overlap with PaaS in the sense that often you need them to implement a PaaS, but how generic they are differs.

Old MPP schedulers for supercomputer clusters and things like VMScluster generic batch queue are probably some of the most obvious intermediate systems between IBM ASP and 21st century. Some proprietary, some open source just niche. They didn't necessarily handle all those higher level ideaas like deployments (closer to basic k8s ReplicationController), often disregarded storage (single system image, cluster-wide common shared mounts, cluster-wide availability of storage in VMScluster, etc.). At the same time High-Availability focused clustering was also building up, but usually with statically allocated resources.

Generally those two branches of clustering started to combine around 2000 (possibly earlier - I am limited here by my own knowledge), with "grid computing" combining ideas of supercomputing scheduling, opportunistic clustering and reliability. It's also around that time that work that ultimately results in in components making up Linux containers starts at Google, soon utilised by Babysitter (long running stable jobs) and Global Work Queue (batch jobs) Not much is published about them. Google starts working on Borg around 2003-2004, basing on the large prior art in scheduling but adapting it to new realities of large node counts in non-HPC tasks. A lot of the work filters into upstreamed kernel components over time, providing the basis for linux containers, with Google developing "LMCTFY" at some point as container runtime. However, they do not perform anything like Docker images.

Google was not the only place that noticed such issues - we had more "cluster specific" and less generic (could be considered transitional technology) cluster managers associated with Hadoop and similar, but also several (still closed, papers published) projects from Microsoft, Alibaba, and others. Some stuff leaks, some is discussed under "how to run those jobs efficiently". 2006 comes and google upstreams some of the kernel features they developed for their system as cgroups. Linux containers project starts in 2008, IIRC clearly inspired by Solaris Zones which debut in 2005. Both still replicate VM instances, if in a "thin" manner (see also OpenVZ, Linux-VServer, BSD Jails).

Amazon EC2 starts in 2006, causing huge change in how one could run their software on the internet. Curiously enough, early days EC2 is mostly stateless, this spurs "12 factor" movement further :) In few years, first custom autoscaling, then built-in AWS ASG (2009) turn up, pushing the approach of pre-baked golden VM images with your software ready to run (further pushing "12 factors", as waiting for scripts to reconfigure the service is costly when you scale up!). Sometime around that time various PaaS services, including Heroku, start building up, requiring ways to schedule customer code across machines. Google App Engine goes GA few months after start of Heroku private beta, with somewhat similar approaches (Google, being Google, is a bit more alien with the custom datastore).

Availability of huge amounts of machines that aren't tightly controlled supercomputers spurs research into clustered resource management, especially since workload-specific designs like those centered around Hadoop are too specific, papers from places running internal proprietary schedulers hint at usability of generic ones. Sometime around 2009? (correct me if I'm wrong, I can't find the start date for Mesos as research project) work starts at UCB for a system that can run diverse job workloads on shared cluster, increasing resource utilization efficiency. I can't find anything reliable on whether the team knows of parallel work at google on borg, but Mesos is probably the first generic system of that class that is open sourced. Uses kernel parts of Borg if unknowingly, doesn't play with packaging. First publications 2010~2011. Twitter becomes one of the first non-research users, running on scale of hundred workers. CloudFoundry starts Warden around that time (unfortunately, Cloud Foundry is very... galapagos-experience for me, so no further details)

2013 is a big year, sees publication of Docker, then still using LXC as backend IIRC, which brings us the image metaphor for packaging. LMCTFY supposedly also goes public around that time (but I don't recall if it was indeed opensourced then, but timeline mostly fits). Oldest presentation discussing cluster management at Google happens at LISA'13 and EuroSys 2013, discussing Omega (not sure if they mention Borg by name). I'm pretty sure the name "borg" is in limited circulation by around that time, but details are still mostly secret. Definite cross-pollination of ideas between different groups at that time, even if most stuff is under NDA ;) Google also publishes "Datacenter as a Computer".

Cambrian explosion of container-based schedulers follows, as people liberate themselves from one-app-per-instance paradigm. (personal aside: I suspect half of it is due to finding Docker horrible in actual deployment, as was my experience in 2014 :D). I'm probably missing details on some (OpenShift pre-3.0?)

Kubernetes starts based on experiences with Borg and Omega, reusing Docker in place of LMCTFY - at least part of the reasoning is obvious, as LMCTFY (and Borg and Omega) had no means to actually provide files to run (a task that was performed by unrelated software at Google). Mesos IIRC didn't originally include anything to distribute applications either. This is (IMHO) the core innovation provided by Docker, integrating packaging into our idea of container.

By 2015 containers moved seriously into public eye instead of being research project or internal implementation detail and adoption takes steam. Kubernetes 1.0 is released, and continues at insane speed (good to read the borg paper and see their pros & cons page, as it relates to this!). Initial set of features doesn't allow replicating Heroku-style PaaS, but IngressControllers debut in 1.4 (maybe 1.3?) based on OpenShift Routes, bringing HTTP routing. Enterprising admins can use it to build a PaaS, but it definitely takes assembly.

Over the next few years tooling for deployment as well as various standard components one might need become more available. It becomes easier to assemble something resembling a PaaS out of kubernetes, but that's because other people write the missing bits (Ingress Controllers, service meshes, logging handlers, etc.). Alternative runtimes build and die in the frothing frenzy of development.

And finally we have last few years, Mesos effectively dying, and most "warehouse-scale computer operating system" tasks ending up incorporating kubernetes, including various PaaS projects moving onto it as base substrate. IMHO that's due to kubernetes API design allowing it to move at speeds that weren't exactly possible for others, and also due to its more generic design (since it came at it from "generic workload scheduler" rather than PaaS angle) meaning that it was applicable in more areas than something designed originally to run a website, for example.

This is amazing! Thanks so much for writing it out. It's so good to read about the same story from a different perspective. I'm only going to respond to one theme in it, I hope that doesn't come across as me not appreciating or not agreeing with what you wrote.

Docker's core innovation from a _technical_ perspective may well be packaging, but I think that really misrepresents the history. Docker, like Heroku, made ordinary developers powerful. It did that through _culture_, by speaking the right language, by knowing what developers want and what they're capable of doing.

I want to talk about Jeff Lindsay[1], the co-author of Docker. He's a prolific open source contributor, a vocal critic of VC funding (and I suspect HN) and passionate advocate of the basic building blocks of software. He created Dokku, introduced Alpine Linux to Docker, founded Flynn, and Glider Labs which innovated things like service discovery and distributed logging in Docker.

Jeff lives and breathes hacker culture. His hallmark is written all over Docker: embrace open source, provide basic building blocks and encourage innovation. He's one of us, a hacker, he builds things for the sheer joy of it. It's obvious he's not in it for the big enterprise money. I mean he's not even it for the reputation, hardly anybody knows his name, everyone just thinks Solomon Hykes created Docker.

These are the people that need to be mentioned in the story, in that documentary. The people that put things into the world, because the world needed them. I don't know the name of the person that came up with The 12 Factors, so it's the same thing, someone knew that the world needed them and just gave them to us.

Of course the story of Omega and Borg is crucial, but they were always motivated by the private interests of an unimaginably large corporation, a corporation that removed its own "Do No Evil" aspiration. Jeff Lindsay has a power that Google doesn't have, the love of software for software's sake. That is a critical part of why we have Docker, and why it became so popular. And Docker's open source status and developer community is an inescapable basis for the world which Kubernetes now rules over.

1. https://dev.to/progrium/hi-im-jeff-lindsay

Now, that's a theme much more interesting from my PoV :)

I don't actually recall that, possibly because I was heavily on the Ops side (sysadmining used to be reasonably easy money, and definitely easier than finding AI job that wasn't ML). From my pov Docker caused an earthshaking change in development process, but it was heavily due to papering over all the crazy between different languages, and of course the infamous "it works on my machine" "then we're shipping your laptop" theme.

I suspect I didn't see this aspect because for me it happened years earlier, when (from my PoV) interest in Rails helped break the monolithic LAMP domination, slowly paving over the divide of those who could afford a dedicated server and those who were stuck with shared LAMP hosting.

By the time Docker landed we had quite extensive suite of tools to maintain complex setups, Puppet was a well known name, 12 factor approaches were big in cloud space, and we had 4 years of heavy evangelism to destroy the wall between Dev and Ops, stopping Dev throwing packages over it and saying "done" while ops cried, and for Ops to get dirty with devs hand to hand. In a way, Docker pushed beyond "development-time only tool" felt like a step back, especially given how early on it was considered pretty normal to run a container, exec into it, modify, then save the result as new image.

Meanwhile, devs tended to not need k8s. Usually they could provision enough locally to run what they needed (possibly in docker containers, making life much easier), but the echoes of the thinking then reverberate IMO in the regular "you don't need k8s because you're not google" message.

Meanwhile on operations side we had the unholy mess that was in no way lessened by Docker, unless you were lucky and you could push your apps into one of those early (hell, I recall a demo written in Bash, I believe) PaaSes building on the simplicity of Docker. Some developers, and less Ops, might have been mollified with that. Meanwhile we had wildly heterogenous systems at times to wrestle together, possibly heavily disparate workloads. Docker looked like a component, but there was always aspect of the amount of work needed to make it generic enough (and I have mentioned few posts ago that running docker in prod was painful, right?). So there was slow acceptance of docker images aas way to handle packaging and distribution easier, and I recall a simple PaaS being demoed by a student literally within days of docker announcement (emulating Heroku and calling it so by name, iirc?), but still, nothing much on the system side. 2013-2014 we were trying to use docker as simpler VM (especially given how hairy ephemeral LXC at the time could be) for running tests of what we would do with automation on "normal" hosts, and reuse of docker as way of dealing with packaging/runtime problem in systems like mesos (but also hadoop/yarn) was becoming more common. It's from this angle, I feel, that Kubernetes arrived on the stage, and this is why I don't necessarily see the same involvement you describe - but that's good, because I also learnt a thing :)

I just don't necessarily feel it was an aspect driving kubernetes, but it definitely feels like it was an aspect driving docker among developers - and with k8s we (more on the ops side ^_-) could deliver such environment for devs in more reliable way.

Sorry for the late reply. I really, really appreciated your in depth replies and your different perspective. Thank you so much for that, I learnt a lot things too :)

Great overview, quite interesting.

Just to add a bit more of history, around 2000, HP starts pushing for HP-UX Vaults as means to also package application into production and restrict what resources they can use from the OS.

Do you have some more links? I did encounter "Vault" themed name in relation to HP-UX, but it sounded more like Trusted Solaris or similar MAC/MLS enabled unix version - but then I had literally a sentence or two for description.

Unfortunately not much much, given how HPE now kind of killed all the nice documents that HP-UX used to have 20 years ago.

This is what I can still find,

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...

Looks like something closer to Solaris Zones (or maybe AIX WPAR?), definitely interesting addition to resource management building block of the journey!