Except k8s is really, really not a PaaS. It can be used to build a PaaS in style of Heroku, but it's not one. It's lineage spans all the way back to IBM ASP in 1967, arguably even to 7094/7044 directly-coupled system (but that didn't have multiple worker nodes), hitting various systems on the way (TOPS-20 Galaxy, VMScluster, HPC scheduling systems for MPP supercomputers and probably many more I can't recall).

It's about efficient use of multiple machines, or as Google & ex-Google employees called it in a book released at similar time as Heroku went from private beta to GA, "Datacenter as a Computer".

12 factors were important, but they just as well can apply to noncontainerised software (arguably, they are just as valid in the ~2000 world of EJB java servers, except replace "environment variables" and the like with JNDI).

That said, Rails exploding in popularity around the same time as EC2 became an option heavily helped push the generic case out of the hell of PHP with occasional CGI (ASP usually meant you had more resources than shared hosting from hell)

I think you're missing my point. We're on HN right, where we often discuss the difference between technical merit and market fit. There's a time and a place for technically correct solutions, which Google excel at, but a successful business needs more than that, it needs to capture a zeitgeist, which Google sometimes fail to do, see Google Plus for example.

Of course the technical lineage of Kubernetes is Borg and parents (which I admit I've never heard of), but I put it to HN that Kubernetes' lineage is not what created the market fit. In fact I think it's disingenuous of the documentary to imply that (in fairness it does give time to Docker), hence why I'm so keen to give my version of events.

Your point about how The 12 Factors apply just as well to non-containerised software is exactly the beauty of them! The 12 Factors don't come from management, they're not an attempt (at least directly) to capture profit from a market. They are somehow fundamental axioms about writing software in this current era, they're maybe even self-evident, universal truths. It turns out that there is a fundamental connection between the ability for a piece of software to be efficiently scaled and for that piece of software to be collaborated on. 2 or more developer machines share essential characteristics with 2 or more scaled containers, which share requirements with 2 or more CI instances ― reproducibility, behaviour guaranteed by tests, ease of downloading, security, configurability through ENV.

You see how we felt like Heroku understood us the humble developer, it made us DevOps, all whilst helping us satisfy the bottom line of managers? Kubernetes is truly an extraordinary piece of software, but it hasn't, nor do I believe it ever will, achieve anything of The 12 Factors' lasting legacy.

It is, in my opinion, at best converged evolution. Maybe because 12 factors are such self-evident truths - though if true, I'd at best credit them as formulation into a listicle of the fundamental behaviour of JCL batch programs on JES2/3 (don't have enough experience with other systems of that class to compare, but I suspect certain commonality).

What my point is that while Heroku might be a very applicable relation to some, it's very explicitly a Platform as a Service, in a way pushing a bit back towards the model that we were escaping from (by invoking Rails name) in mid-2000 to cheap full-OS machines, from the confines "PaaSsy" LAMP.

Kubernetes didn't capture a zeitgeist by introducing any of the features (though it definitely pushed containers up into enterprise world a lot). It was in no way first, or even second, despite coming from home of linux containers. It doesn't even force you to use 12 factors - I should know something about it, having done what many called stupid, as in I did "lift and shift" of over 60 "classical" applications that in no way fit 12 factors onto it.

Now, trodding a bit further away from concrete facts and into more a world of personal opinion...

Kubernetes captured the zeitgeist by working. There was immense desire across all kinds of business, developments, etc. for something that fit nebulous label of "private cloud". Often with local deployments involved. And to do it cheaper (or at least, easier to automate) than existing "heavy" virtualisation stacks. Not everyone could use at all, or just burn money on AWS, many VPS providers were more like classic VMware virtualization clusters in experience. Some had money to burn on Heroku and actually fit running on it (or were lucky to fit in the cheap band of usage).

Thus you had OpenStack and the dwarfs: CloudStack, Eucalyptus, OpenNebula, Ganeti, with more adventurous ones building on Mesos or fully custom.

All tackling the problem of, essentially, providing simpler and faster (in turnaround time) way of handling users access to resources.

The most moneyed and successful (at least from pov of Ops) was OpenStack. And it was a hot mess. Upgrading an openstack cluster was a meme, and not a nice one. Meanwhile containers gain steam, for good or bad, and people start to move from moving from the quite bad "single app VM instances", to containerised approach (12 factors already hit harder due to the single-app VM approach popularised by AWS ASGs coupled with cloud-init).

Into this enters kubernetes. Yes, it might seem hard to setup - but it's no OpenStack, it brings no licensing purchasing mess of VMware, it initially kills one of the more annoying aspects (though it will return) which is the overlay networking. It brings unified API, simple operating model (wish I kept the old 1.0-1.4 presentations on hand that went bottom up!). We can deploy stateless services in somewhat sensible way (and if you know how, they can be stateful too). The saner internal design and API commonality can be arguably compared to the effect of the original "heroku" command that allowed you to just deploy from a git repo - just from the point of Ops team.

By next year, we move from simple ReplicationController to ReplicaSets + Deployments, as well as gaining Ingress beta and first rush of "pluggable" componentry in the form of Ingress Controllers. It's arguably the time where you can start dealing with K8s as simplistic replacement for Heroku - except it's not some closed magic operated by the Wizard of Oz, but something you can deploy on bunch of Raspberry Pi in middle of presentation at a conference after suffering a long trip (we even forgot, iirc, to bring our planned kube-master machine and had to improvise). Early versions of TPRs (priors of CRDs) make me give what were probably some of the first talks about operators (before CoreOS people coined the term) - but I used the same terminology as K8s used (Controllers).

By 2017, the recommended way to deploy openstack starts to be "deploy openstack on top of k8s", and DC/OS starts to visibly lose. K8s makes certain things forced upon you, but you find that it doesn't matter and tends to make sense (Pods having one IP, etc.). Secondary/Tertiary adoption follows, bad memes start about kubernetes (like you only need it for scale, that it's cloud specific, etc. etc.) but it doesn't really matter. Soon, operators start flowing in, thanks to simplicity of the API model (this evolves later to things like Crossplane and even separating kube-apiserver into standalone program, to run without rest of k8s). K8s manifests might feel like JCL, but you don't care because of how bloody more complex the same things used to be before.

Heroku might have provided easy way to deploy constrained application for a developer, but, at least in my experience, k8s became the lever with which I could move a lot of servers with little effort, whether to run large batch jobs, complex meshes of microservices, CI build jobs, or build a heroku-style PaaS. As developers learnt to follo 12 factors, the job became easier.

P.S. Funnily enough, probably because accidents of history kept me on the Ops side, but IME kubernetes adoption always had push from the sysadmin side, not necessarily developer side.

P.P.S. As nice as Heroku is, there's a reason why it's my main example of burning money, especially contrasting how it's billed as "cheaper than paying for an Ops engineer". In the world of global pricing, what works as cheap in Silicon Valley is... not cheap in most of the world.

P.P.P.S This kinda went ranty, I guess I should maybe write a blog post on this perspective...

Very interesting about JCL being an older precedent for the 12 Factors, I've never used it, so I'll have to take a look.

I totally know what you mean about OpenStack! So much potential, but just way too complicated. But I'm not sure it's fair to say that Kubernetes was the first project to Just Work™. I never tried Mesos, but it's always seemed more accessible to me. But I did use Deis and Flynn though, and they were a delight in comparison. Deis of course being the home of Helm, Kubernetes' now de facto packager manager.

Also don't you think that Kubernetes explicitly embraced Docker right from the start as a first-class citizen? OpenStack only ever bolted it on as an after thought. Kubernetes knew what the buzz was and made sure to let everyone know that Docker was front and centre. This is why I think it's better to think of Kubernetes as a spiritual child of Docker (and therefore everything that came before Docker) rather than a technical descendant of Borg.

Although Deis and Flynn were certainly already creaking under the weight of what it truly meant to support the full gamut of cloud features, they were heading in the right direction. Google really did knock it out of the park, which I'm grateful for, but I think they did it more with the sheer weight of money and reputation than any innovative technology. Nobody really knew the devs behind Deis and Flynn, but everyone knows Google's reputation. And so we all just came to warm our hands around Kubernetes' fire, and the rest is history.

JCL and JES2/3 provided a bunch of features (of course, very primitive compared to anything newer) that helped run programs in a world before expectations of "wasting precious disk space" on configuration or state that wasn't critical, thus it tended to operate remarkably similar to running 12 factor batch jobs - for example, you would use Data Definition (origin of Unix dd command - it was a joke on JCL syntax, iirc) to specify open files (with named handles) passed to a program (consider passing that through environment variables or parameters in 12 factor app), telling it to run an otherwise stateless binary, and possibly embedding a short command program (interpreted by it) in JCL stream. A program might be preinstalled on disk, or pulled from tape, your data might be mostly on tapes, etc., and your JCL on a bunch of cards that you submit is, effectively, your declaration of environment for it and the system operators take care it's run properly - but outside of that JCL, there might be no other state involved.

Kubernetes might seem to embrace Docker, but in reality it hid the Docker. In fact, kubernetes explicitly avoided using features from docker outside of certain basics - I'd even say it's part of what I liked the most, as I found Docker to be incredibly clunky once past "local development" stage. As early as 2016 we had alternative runtimes that took in docker images (and sometimes not even them) and ran them in completely different matter. It was just a question of time, when Docker will be eliminated from k8s stack, rather than if. The only real thing that k8s took from Docker and not somewhere else was the idea of container images as packaging format (containers having been widely used at google with LMCTFY and other runtimes before, and LXC predating Docker in wider world with even primitive package system).

Mesos had the issue that, at least in my experience, every bit of the stack was separate. Running "just Mesos" wasn't really an option, and every framework seemed to be building with its own APIs, creating a disconnected mess (mind you, my first actual use of Mesos came after already using k8s for a while). What comes as common bits (without operators) in k8s 1.4, involved IIRC 2-3 different frameworks with disjoint APIs on DC/OS. Quite a different experience, even if you used DC/OS which packaged it altogether for you. Mind you, DC/OS arguably also totally killed community around Mesos in my PoV, but that's a question to someone deeper involved in that ecosystem.

As for OpenStack vs. K8s... I can't even compare the scale difference of running "reasonably operational" k8s cluster vs. "openstack that isn't just devstack on dev's laptop". And when it comes to upgrading it really becomes unfunny.

Also, at least early on, it really didn't seem that K8s would become the half ton gorilla it is now, and while people might have not known devs behind Deis and Flynn, it was in my experience heavily the evangelism of early adopters that pushed it, later bolstered by GKE becoming a thing. Despite that and google's supposed reputation (they really, really don't know how to monetize it, instead burning it - otherwise GCP would be much more of a success commercially than it is, not third in public perception of people I talk with), alternative runtimes continued on and people still believed that DC/OS might be the winner, or that a bunch of other platforms might maybe standardize some bits around docker but no clear winner might appear. Admittedly I haven't heard of Flynn, but I do recall Deis, pre-k8s Rancher, Docker Swarm, CoreOS' Fleet, the unlamented OpenStack thingy, and others being regularly discussed.

The one thing that IMO made k8s truly groundbreaking was the kube-apiserver design, from first principles. Yes, a whole cluster is complex, but it builds from simple component ideas, with the simple patterns of apiserver tying everything together (said "simplicity" is grounded thoroughly in control theory and, I wonder if anyone noticed, symbolic AI designs like Blackboard Systems). This meant that evolving that system, first all internally through combined controller binary, then through TPRs and finally CRDs, was easy. Outwardly (there's some extension if you look close) the base k8s API didn't change since version 1.0, despite huge changes in what you could do with it. Something that I don't think any other alternative took care of, with exception of maybe generalized look at AWS APIs (if you squint to blur the inconsistency and try not to notice gaps hinting at private APIs and differences between what's doable from Console and API).

A bunch of that came explicitly from having experience of much more haphazardly designed Borg APIs and their ossification, and it sometimes showed in discussions on github repo (I recall early requests for CPU pinning being refused by understandably vague but useful explanation of how it led to broken design in borg api and that they would recommend shelving till a good interface gets designed)

Tapes and cards!! OMG that puts JCL into perspective!

Sure Kubernetes' API hid Docker, but its marketing certainly didn't. And I'm not sure it's right to say Docker was eliminated from Kubernetes, it was simply made optional. I think it's hard to argue that Docker isn't still the pioneer, like I mentioned Docker is the OG market fit which all the other runtimes are trying to emulate.

So well-designed APIs, which Mesos doesn't have, I can believe that. But I think it's fair to say Flynn and Deis put a lot of effort into theirs. I forgot about Rancher and Fleet! I don't know about their APIs. For sure Flynn and Deis never got as far as CRDs, but they still achieved those basic concepts we use in Kubernetes today; deployments, storage, config, logs, routes, networks, etc.

I'm just still not convinced that Borg did anything new or innovative that wasn't already in some Open Source project before Kubernetes came along.

The EuroSys paper on Borg is a good place to get the background on Borg, how it influenced Kube, and how it relates to other systems: https://research.google/pubs/pub43438/

Well, the issue is that Borg is not public, and isn't direct ancestor of kubernetes - a bunch of design decisions were informed by the Omega project which was an attempt to build a new Borg (for example, pluggable schedulers, a rather core aspect of k8s).

Marketing hanging onto Docker was a great marketing move, even if all that docker brought was essentially images.

As for basic concepts - I believe that the core issue is that Flynn/Deis/Heroku converge from the opposite starting point (it's arguably a common case for all PaaS).

A PaaS wants to provide those high-level building blocks to the end user, but (especially the case with Heroku and other closed source examples), it hides everything else. In a way it is an offshoot of, if much better designed usually, of the classic LAMP experience, except usually with some extras provided - and these days we get APIs instead of crappy web panel (or cPanel, in fact ^_-).

The implementation details of them aren't for end user consumption, and generally the less they leak the better.

OTOH, we have infrastucture-side systems, which start with "I have X resources and I want to utilise them efficiently". Those tend to overlap with PaaS in the sense that often you need them to implement a PaaS, but how generic they are differs.

Old MPP schedulers for supercomputer clusters and things like VMScluster generic batch queue are probably some of the most obvious intermediate systems between IBM ASP and 21st century. Some proprietary, some open source just niche. They didn't necessarily handle all those higher level ideaas like deployments (closer to basic k8s ReplicationController), often disregarded storage (single system image, cluster-wide common shared mounts, cluster-wide availability of storage in VMScluster, etc.). At the same time High-Availability focused clustering was also building up, but usually with statically allocated resources.

Generally those two branches of clustering started to combine around 2000 (possibly earlier - I am limited here by my own knowledge), with "grid computing" combining ideas of supercomputing scheduling, opportunistic clustering and reliability. It's also around that time that work that ultimately results in in components making up Linux containers starts at Google, soon utilised by Babysitter (long running stable jobs) and Global Work Queue (batch jobs) Not much is published about them. Google starts working on Borg around 2003-2004, basing on the large prior art in scheduling but adapting it to new realities of large node counts in non-HPC tasks. A lot of the work filters into upstreamed kernel components over time, providing the basis for linux containers, with Google developing "LMCTFY" at some point as container runtime. However, they do not perform anything like Docker images.

Google was not the only place that noticed such issues - we had more "cluster specific" and less generic (could be considered transitional technology) cluster managers associated with Hadoop and similar, but also several (still closed, papers published) projects from Microsoft, Alibaba, and others. Some stuff leaks, some is discussed under "how to run those jobs efficiently". 2006 comes and google upstreams some of the kernel features they developed for their system as cgroups. Linux containers project starts in 2008, IIRC clearly inspired by Solaris Zones which debut in 2005. Both still replicate VM instances, if in a "thin" manner (see also OpenVZ, Linux-VServer, BSD Jails).

Amazon EC2 starts in 2006, causing huge change in how one could run their software on the internet. Curiously enough, early days EC2 is mostly stateless, this spurs "12 factor" movement further :) In few years, first custom autoscaling, then built-in AWS ASG (2009) turn up, pushing the approach of pre-baked golden VM images with your software ready to run (further pushing "12 factors", as waiting for scripts to reconfigure the service is costly when you scale up!). Sometime around that time various PaaS services, including Heroku, start building up, requiring ways to schedule customer code across machines. Google App Engine goes GA few months after start of Heroku private beta, with somewhat similar approaches (Google, being Google, is a bit more alien with the custom datastore).

Availability of huge amounts of machines that aren't tightly controlled supercomputers spurs research into clustered resource management, especially since workload-specific designs like those centered around Hadoop are too specific, papers from places running internal proprietary schedulers hint at usability of generic ones. Sometime around 2009? (correct me if I'm wrong, I can't find the start date for Mesos as research project) work starts at UCB for a system that can run diverse job workloads on shared cluster, increasing resource utilization efficiency. I can't find anything reliable on whether the team knows of parallel work at google on borg, but Mesos is probably the first generic system of that class that is open sourced. Uses kernel parts of Borg if unknowingly, doesn't play with packaging. First publications 2010~2011. Twitter becomes one of the first non-research users, running on scale of hundred workers. CloudFoundry starts Warden around that time (unfortunately, Cloud Foundry is very... galapagos-experience for me, so no further details)

2013 is a big year, sees publication of Docker, then still using LXC as backend IIRC, which brings us the image metaphor for packaging. LMCTFY supposedly also goes public around that time (but I don't recall if it was indeed opensourced then, but timeline mostly fits). Oldest presentation discussing cluster management at Google happens at LISA'13 and EuroSys 2013, discussing Omega (not sure if they mention Borg by name). I'm pretty sure the name "borg" is in limited circulation by around that time, but details are still mostly secret. Definite cross-pollination of ideas between different groups at that time, even if most stuff is under NDA ;) Google also publishes "Datacenter as a Computer".

Cambrian explosion of container-based schedulers follows, as people liberate themselves from one-app-per-instance paradigm. (personal aside: I suspect half of it is due to finding Docker horrible in actual deployment, as was my experience in 2014 :D). I'm probably missing details on some (OpenShift pre-3.0?)

Kubernetes starts based on experiences with Borg and Omega, reusing Docker in place of LMCTFY - at least part of the reasoning is obvious, as LMCTFY (and Borg and Omega) had no means to actually provide files to run (a task that was performed by unrelated software at Google). Mesos IIRC didn't originally include anything to distribute applications either. This is (IMHO) the core innovation provided by Docker, integrating packaging into our idea of container.

By 2015 containers moved seriously into public eye instead of being research project or internal implementation detail and adoption takes steam. Kubernetes 1.0 is released, and continues at insane speed (good to read the borg paper and see their pros & cons page, as it relates to this!). Initial set of features doesn't allow replicating Heroku-style PaaS, but IngressControllers debut in 1.4 (maybe 1.3?) based on OpenShift Routes, bringing HTTP routing. Enterprising admins can use it to build a PaaS, but it definitely takes assembly.

Over the next few years tooling for deployment as well as various standard components one might need become more available. It becomes easier to assemble something resembling a PaaS out of kubernetes, but that's because other people write the missing bits (Ingress Controllers, service meshes, logging handlers, etc.). Alternative runtimes build and die in the frothing frenzy of development.

And finally we have last few years, Mesos effectively dying, and most "warehouse-scale computer operating system" tasks ending up incorporating kubernetes, including various PaaS projects moving onto it as base substrate. IMHO that's due to kubernetes API design allowing it to move at speeds that weren't exactly possible for others, and also due to its more generic design (since it came at it from "generic workload scheduler" rather than PaaS angle) meaning that it was applicable in more areas than something designed originally to run a website, for example.

This is amazing! Thanks so much for writing it out. It's so good to read about the same story from a different perspective. I'm only going to respond to one theme in it, I hope that doesn't come across as me not appreciating or not agreeing with what you wrote.

Docker's core innovation from a _technical_ perspective may well be packaging, but I think that really misrepresents the history. Docker, like Heroku, made ordinary developers powerful. It did that through _culture_, by speaking the right language, by knowing what developers want and what they're capable of doing.

I want to talk about Jeff Lindsay[1], the co-author of Docker. He's a prolific open source contributor, a vocal critic of VC funding (and I suspect HN) and passionate advocate of the basic building blocks of software. He created Dokku, introduced Alpine Linux to Docker, founded Flynn, and Glider Labs which innovated things like service discovery and distributed logging in Docker.

Jeff lives and breathes hacker culture. His hallmark is written all over Docker: embrace open source, provide basic building blocks and encourage innovation. He's one of us, a hacker, he builds things for the sheer joy of it. It's obvious he's not in it for the big enterprise money. I mean he's not even it for the reputation, hardly anybody knows his name, everyone just thinks Solomon Hykes created Docker.

These are the people that need to be mentioned in the story, in that documentary. The people that put things into the world, because the world needed them. I don't know the name of the person that came up with The 12 Factors, so it's the same thing, someone knew that the world needed them and just gave them to us.

Of course the story of Omega and Borg is crucial, but they were always motivated by the private interests of an unimaginably large corporation, a corporation that removed its own "Do No Evil" aspiration. Jeff Lindsay has a power that Google doesn't have, the love of software for software's sake. That is a critical part of why we have Docker, and why it became so popular. And Docker's open source status and developer community is an inescapable basis for the world which Kubernetes now rules over.

1. https://dev.to/progrium/hi-im-jeff-lindsay

Now, that's a theme much more interesting from my PoV :)

I don't actually recall that, possibly because I was heavily on the Ops side (sysadmining used to be reasonably easy money, and definitely easier than finding AI job that wasn't ML). From my pov Docker caused an earthshaking change in development process, but it was heavily due to papering over all the crazy between different languages, and of course the infamous "it works on my machine" "then we're shipping your laptop" theme.

I suspect I didn't see this aspect because for me it happened years earlier, when (from my PoV) interest in Rails helped break the monolithic LAMP domination, slowly paving over the divide of those who could afford a dedicated server and those who were stuck with shared LAMP hosting.

By the time Docker landed we had quite extensive suite of tools to maintain complex setups, Puppet was a well known name, 12 factor approaches were big in cloud space, and we had 4 years of heavy evangelism to destroy the wall between Dev and Ops, stopping Dev throwing packages over it and saying "done" while ops cried, and for Ops to get dirty with devs hand to hand. In a way, Docker pushed beyond "development-time only tool" felt like a step back, especially given how early on it was considered pretty normal to run a container, exec into it, modify, then save the result as new image.

Meanwhile, devs tended to not need k8s. Usually they could provision enough locally to run what they needed (possibly in docker containers, making life much easier), but the echoes of the thinking then reverberate IMO in the regular "you don't need k8s because you're not google" message.

Meanwhile on operations side we had the unholy mess that was in no way lessened by Docker, unless you were lucky and you could push your apps into one of those early (hell, I recall a demo written in Bash, I believe) PaaSes building on the simplicity of Docker. Some developers, and less Ops, might have been mollified with that. Meanwhile we had wildly heterogenous systems at times to wrestle together, possibly heavily disparate workloads. Docker looked like a component, but there was always aspect of the amount of work needed to make it generic enough (and I have mentioned few posts ago that running docker in prod was painful, right?). So there was slow acceptance of docker images aas way to handle packaging and distribution easier, and I recall a simple PaaS being demoed by a student literally within days of docker announcement (emulating Heroku and calling it so by name, iirc?), but still, nothing much on the system side. 2013-2014 we were trying to use docker as simpler VM (especially given how hairy ephemeral LXC at the time could be) for running tests of what we would do with automation on "normal" hosts, and reuse of docker as way of dealing with packaging/runtime problem in systems like mesos (but also hadoop/yarn) was becoming more common. It's from this angle, I feel, that Kubernetes arrived on the stage, and this is why I don't necessarily see the same involvement you describe - but that's good, because I also learnt a thing :)

I just don't necessarily feel it was an aspect driving kubernetes, but it definitely feels like it was an aspect driving docker among developers - and with k8s we (more on the ops side ^_-) could deliver such environment for devs in more reliable way.

Sorry for the late reply. I really, really appreciated your in depth replies and your different perspective. Thank you so much for that, I learnt a lot things too :)

Great overview, quite interesting.

Just to add a bit more of history, around 2000, HP starts pushing for HP-UX Vaults as means to also package application into production and restrict what resources they can use from the OS.

Do you have some more links? I did encounter "Vault" themed name in relation to HP-UX, but it sounded more like Trusted Solaris or similar MAC/MLS enabled unix version - but then I had literally a sentence or two for description.

Unfortunately not much much, given how HPE now kind of killed all the nice documents that HP-UX used to have 20 years ago.

This is what I can still find,

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...

Looks like something closer to Solaris Zones (or maybe AIX WPAR?), definitely interesting addition to resource management building block of the journey!

This is an interesting story but it’s nothing more than fan fiction. Please provide some reason for your assertions.

While the entire comment could be contradicted, a quick example is that google’s SRE book details much more developer strategies, including from a massive scale and enterprise perspective, than 12 factor web app. This book was assembled but industry experts with experience from Borg, with borg emerging from the global work queue — decades before heroku existed.

> google’s SRE book details much more developer strategies [...] than 12 factor web app

I would argue that's precisely the problem. Google are clearly capable of doing complex things, but inspiring and nurturing an entire cultural shift is a different category of "complexity", which in the story of Paas, Google did not achieve, they merely exploited it.

You seem to be arguing against yourself here though. Kubernetes is successful, and it doesn't precisely fit the mold of the 12 factor app. It fits an overlapping but distinct, google style, development model. They successfully achieved that shift.

I think you're arguing that that's a bad thing. But it's a thing that very much did happen, because kubernetes is successful.

My thesis isn't that Kubernetes isn't good, nor that it isn't successful (I use it, recommend it and am deeply grateful for it). Rather I'm disappointed that the documentary doesn't pay respects to the milieu that primed the world for Kubernetes, which to me makes it seem like Google also inspired the mindset and demand for something like Kubernetes, which I personally believe is inaccurate and more importantly unfair on the many innovative projects upon whose shoulders Kubernetes stands.

Didn't Google's Borg precede Heroku by years? Surely its unfair to claim that they "brought little more to the table than money and influence"

I think it's clear in everyone's minds that right from the start Kubernetes was making Docker a first class citizen right? Borg did not use Docker. Borg may well be the technical ancestor of Kubernetes, but Kubernetes owes significantly more to the paradigm shift and cultural revolution that Docker brought. Therefore I believe it's more relevant to consider Docker the inspiration and ancestor of Kubernetes.

Borg didn't use docker containers, but does use containers, they're just a form of containers that are propriety and predate docker by a decade (but like they're functionally really similar to docker containers, at least from the perspective of the orchestration layer).

So docker is maybe the thing that allowed kubernetes to catch on, but that's just because the external world was catching up to what borg had been doing for more than a decade, and so kubernetes was Google sticking a decade or more of expertise in container orchestration into the OSS world.

I just don't think it's fair to define Docker by merely its ability to isolate resources (that which distinguishes containers from VMs). Docker is an entire ecosystem of best practices, standards, tooling, packaging, documentation and community support.

Google caught up to Docker not the other way round.

> Docker is an entire ecosystem of best practices, standards, tooling, packaging, documentation and community support.

When kubernetes was released, the majority of that ecosystem didn't yet exist. Kubernetes influenced many of those best practices, standards, and tooling. So in shaping the ecosystem, yes, docker was catching up to Google. Like I think this would be a reasonable argument if Kubernetes was released today, but it was released only about a year after Docker was.

(heck, iirc, even docker as a tool was originally built on LXC, which I think google functionally upstreamed as part of borg hypervisor stuff, but I could be mistaken there. I do know that LXC was based on Google work though: https://www.nextplatform.com/2016/03/22/decade-container-con...)

Edit: Ok yeah I think I figured it out, cgroups is the borg upstreamed feature (https://news.ycombinator.com/item?id=9506800), LXC and evenutally docker are built atop cgroups, etc. etc. So the entire modern containerization infra is built atop a feature borg upstreamed.

We might be going in circles a bit here. Sorry if I'm misunderstanding something. Docker was released with Dockerfiles, registry support (maybe not Docker Hub, I don't remember), a generous CLI; `start`, `build`, `ps`, etc. So I would very much say the majority of the ecosystem existed before Kubernetes.

Yes LXC and cgroups were Google contributions before the existence of Docker. But jails and containers in general were also general ideas before LXC and cgroups, I think most notably in BSD.

I feel like we must have had different experiences of that period 8 or so years ago. I distinctly remember the hype and bandwagon feeling of Docker's release. I remember various projects trying to capitalise on Docker to make PaaS-type projects; Swarm, Deis, Flynn (even my own Peas project), before Kubernetes came along. It seems so clear to me that Google were reactive rather than proactive. I really don't think Kubernetes would be a thing without Docker.

I don't really dispute much you say here (although I'd argue that calling Dockers API an "ecosystem" and "best practices" is a stretch, do you write your dockerfiles the same way you did today as in 2013?). Like yes, k8s had to come after docker. What I dispute is how any of that is the spiritual successor of heroku (or GAE or any other PaaS).

And even then I think you understimate googles contribution of "we've literally been doing container orchestration for longer than anyone else, here's tooling that is built with lessons we've learned in mind". That's far more than just "money and influence", and I don't think it's "exploitative".

That's a very fair criticism of my use of "exploited". And also I accept that "ecosystem" could be a descriptive stretch for Docker's release day.

I think in the same way that you're concerned I'm misrepresenting Google, I'm concerned that the documentary misrepresents the landscape of developers and projects that set the scene for Google's Kubernetes. Yes Google made significant contributions to the story that led up to Kubernetes, but crucially, I believe they didn't nurture the culture, they didn't create the mindset that makes developers (and managers) ready to think in a containerised way. And I think that's important because the smaller voices that did make those innovations easily get drowned out by the might of Google.

Containers as concept predate BSD stuff, HP-UX and Tru64 were two UNIXes that had them first, following on previous enterprise computing platforms.

Not really. It's not obvious from the outside, but from a Google developer POV borg (and workqueue mostly) had everything docker had. It just achieved it differently (monrepo, standard build, custom name service, etc).

Google pre-k8s achieved "container" orchestration by controlling the whole stack in detail. K8S provided a way to get most of that with mostly arbitrary code.

You're really missing my point. It's not always enough to create a technically successful project. Docker was a _cultural_ revolution, it had its finger on the pulse, and without that revolution Kubernetes would not have the traction it does.

Indeed. The striking difference is that Google's internal culture is (?used to be, at least at the time this documentary takes place) that of: build your static binaries, perhaps bundle a couple of other files in a package (basically a tarball) if you really must, and run in the code in a "container" whose system image is not controlled by the author of the code package but by the place where you deploy the code (the compute infrastructure, ime. the Borg cell). When the admins of the compute infrastructure decide to roll out a new system image they'll let you know and you'll have to coordinate in order to maintain proper hygiene.

The public version of this embraced a fundamentally different status quo: software is a mess. People pull in dependency in many different ways. You may end up using some python code that has a C extension that in order to build requires some C library whose installation instructions are a mix of run some install script after having installed some apt package.

This idea that you build a container with a container (and accepting that the resulting mess is "ok") is indeed something that has to be granted as a form of innovation to docker. It created a whole new industry of people trying to tame the dependency hell and supply chain woes that have been just swept under a carpet.

Linux containers were important, but Docker gets a lot of credit for popularizing them. Even still, Kubernetes makes containers dramatically more useful than just running a bunch of Docker containers on EC2 instances or whatever. Not only does it schedule containers, but it enables controllers and operators for enabling increasingly higher levels of abstraction. I don’t think Heroku has any analog (nor should it because Heroku is a PaaS).

But there were/are many projects that offer those higher levels of abstraction on top of Docker; Deis, Flynn, Rancher, Fleet. I just don't think Kubernetes was that innovative in that regard. It won out because I mean of course it's a fantastic piece of software, that's a given, but the differentiating factor is Google's brand, not any particular technical novelty.

I think “primed the world” is a much more defensible position than what I originally thought you were arguing (that Heroku influenced Kubernetes directly).

Sorry if I caused any confusion, that's exactly my point. I didn't mean to imply that Heroku directly influenced the design and architecture of Kubernetes.

Decades before Heroku existed? Can you back that up?

FWIW — we never viewed Kubernetes as a PaaS. We were informed by both Borg and GAE (and Heroku and PCF).

The pattern that you see often with PaaS is that they work great until they don’t. When you want to do something that doesn’t fit the patterns that the PaaS was built for you are out of luck and pretty much have to drop down to VMs.

We aimed to have k8s be something that was in the middle. Generic enough to work with a large number of workloads while providing significant advantages above raw VMs (and a level of consistency across infra providers).

The tradeoff is that it is more complicated than the typical PaaS. That is the tradeoff. The goal then was to build kubernetes to be built upon. We never claimed to be solving the whole problem in one system.

This is similar, in some ways, to the design of Mesos. But to do something new with Mesos requires an entirely different “framework” that exposes its own API. You see this with things like Marathon or Aurora or Chronos. While the underlying scheduler was shared across these there was no common API and so it was difficult for users to mix/match across these.

By focusing on a consistent IaaS-ish API for k8s (and eventually, with CRDs, building extension mechanisms) we aimed to create an ecosystem of things building on k8s.

Whereas PaaS are often a framework where you put your code in and a lot of decisions were made for you, I always viewed k8s as a toolkit where you get a lot of powertools that you can use in various ways. It is a very different philosophy.

12 factor and Docker obviously play a big part of this. And those built on experience and systems that came before them. We are all standing on the shoulders of those that came before us.

For me, this API design experience was informed by my time at Microsoft. Things like VB and IE/web were very much, at that time a framework (the web is turning into a toolkit over time now). I saw, multiple times, how those things were super useful but were limited. Things like the CLR and the C# ecosystem evolved the VB model to be much more of a toolkit.

We see this also in the Java world. I’m not an expert there, but the move from app servers to things like Spring Boot also show the shift from framework to toolkit.

AWS itself is a toolkit. And IMO, the difference between something like lambda/faas and a PaaS is that lambda/faas only works when it is in context of the larger toolkit that is IaaS cloud.

(I’m one of the founders of k8s and ended up with quite a bit of screen time in the documentary. My only regret is that more people that were so important in the k8s community couldn’t be featured!)

Thanks for responding to my soap boxy opinions!

I actually wrote my own complete PaaS from scratch once[1], and contributed to a number of other PaaS projects. I totally agree that PaaS projects work great until they don't. The reality is that a PaaS has to also be a Iaas, and yes Kubernetes really hits that middle ground. Thank you so much for your part in that.

1. https://github.com/tombh/peas

> We see this also in the Java world. I’m not an expert there, but the move from app servers to things like Spring Boot also show the shift from framework to toolkit.

Having ridden the heyday of Java application servers, I see Kubernetes as LanguageAgnosticEE, just way more complex than dealing with WebSphere or WebLogic ever was.

Arguably its inherent complexity given that it's further toward bare infrastructure from them, just from supporting other languages.

Thanks for your effort on k8s, as well as for taking the time to contribute to this discussion.

> My only regret is that more people that were so important in the k8s community couldn’t be featured!

Would you like to list some of those individuals here? Perhaps they have homepages/blogs we can check out?

I hesitate to start listing names as I'm sure that I'd accidentally leave someone out. Suffice to say that so many folks had an amazing impact and I'm grateful to all of them for being part of it.

Fair enough - and thank you for taking the time to reply.

I've been involved in multiple migrations to public cloud infrastructure (some were unnecessary, but such are the trends...), and without failure, the 12 factors manifesto shows up in the very first days of the discussions as the first target to reach before anything else is doable.

I very much enjoy it and I'm glad whether I see someone mention it or I can spread and teach its wisdom.

> The 12 Factors aren't just about pushing your app on Kubernetes or whatever, they're just as much, if not more so, ideas that make _you_ a better engineer.

This is absolutely true, you can build a well factored app with intent to run it on 1 server. I would say it's very much worth it because adhering to these best practices will not only make maintenance of your app easier but it'll make things SO much easier to move from 1 server to multiple servers later on if you need to where "servers" in this case could be a Kubernetes cluster, Heroku or anything in between.

If anyone is curious I recently put out a 90 minute ad-free video going over how to create a 12 factor app with a Dockerized app at https://nickjanetakis.com/blog/creating-a-twelve-factor-app-.... It goes over the whole original 12 factor guide from start to finish and applies it to a code base.

Even now 10+ years after the 12 factor guide was originally released it still very much applies today because the guidelines transcend technology.

I remember reading this guide so many years ago and thinking "yep, this is it, they nailed it". So much goodness has come out of the Ruby community, especially around the idea of creating high quality services and tools built from really successful applications. In this case, the 12 factor guide is an extraction of best practices after running hundreds of thousands of apps on Heroku and Rails is an extraction of building Basecamp.

I think you're very deeply immersed in the Ruby/Heroku world and projecting that onto Kubernetes and everything else. Kubernetes' derives from Google's lengthy experience with Borg, which almost certainly predates Heroku. I doubt very much that Google was steeped in Heroku/Ruby culture. Yes, Heroku dynos are similar to linux containers, but containerization (the idea of binning multiple applications onto a single operating system instance) predates heroku by at least 6 years and the idea of binning multiple applications onto a single hardware instance goes back much further.

Moreover, Kubernetes and Heroku aren't even in the same class of tools. Kubernetes is an orchestrator for containers, which obviously can be used as the foundation for a PaaS, but Kubernetes isn't itself a PaaS even when you consider the various managed Kubernetes offerings which include some higher level abstractions built atop Kubernetes primitives.

Further still, it's not like Google marketed Kubernetes to any significant degree prior to its own adoption of Kubernetes as its official container orchestration offering for its cloud (which was after the writing had been on the wall for a while that Kubernetes would become the de facto container orchestrator).

TL;DR: Just because other tools and technologies predated Kubernetes didn't imply that it was significantly impacted by them.

I haven't written Ruby nor used Heroku in many years. I think it's more relevant to say I'm very deeply immersed in Open Source. I contributed to OpenRuko[1], Deis, Flynn and even wrote my own PaaS, Peas[2] that got on the front page of HN. I've seen first hand the journey that PaaS has taken. Maybe we're getting hung up on the strict definition of PaaS here. Let's just say the thing that lets us easily scale a startup idea, like Uber, Airbnb etc. Sure the dyno/docker is the core component, but we inevitably have to deal with state, Redis, Postgres etc as well. In that sense Kubernetes is very much the Open Source version of Heroku.

I think it's fairer to say that Google _repurposed_ Borg, in order not to be left out of the changing landscape. Borg may be the technical predecessor to Kubernetes, but my point is that Borg is very much not the spiritual ancestor of the new containerised paradigm.

1. https://github.com/openruko

2. https://github.com/tombh/peas

I would be very interested to hear why Heroku and not Borg is the spiritual ancestor of Kubernetes because that claim is contrary to everything I’ve read and listened to on the matter.

I'd hoped that's exactly what comes across in my top-level comment. I've been lucky enough to be at or near the coalface of PaaS for most of its life, both as a basic engineer getting my head around deploying CRUD apps on Heroku and its ilk, and as an architect contemplating the designs of various PaaSs. My little sketch of a story is how I saw things unfold. Don't get me wrong, Kubernetes is amazing, it's everything we hoped for, I use it and recommend it. But the only thing that's innovative about it is that it's backed by a big player, so we can rely on at least a few years of active development and mindshare. But from my perspective, the great innovation, the thing that really changed the game, was The 12 Factor App manifesto. That's what deserves a documentary made about it, because it's just a text document, but it inspired so much.

And arguably borg and global work queue are predated by generic batch queue of VMScluster (and possibly TOPS-20 Galaxy, but I haven't played with those) and those are predated by IBM Job Entry Subsystem 3, which even operated in similar manner to k8s with "master" mainframe, usually a lower power model, scheduling jobs on multiple "worker" mainframes.

I think GAE (Google App Engine) was in production before Heroku got anything out there...

GAE was also a hero, but too early for non-Googler.

Yes they were. But like I say, I think the 12 Factors were more important. And Heroku in general just understood hacker culture better.

Scale - How Cloud Computing took over the World.

(This is describing a book that would be interesting to read).

Chapter 1 - Cobol on the Mainframe

A description of how the world's financial infrastructure ran on mainframes from the 1970s onward.

Chapter 2 - The rise of the Internet

A quick description of the take off the internet and how companies would scale on the internet. Ops teams of hundreds and Solaris machines in data centers.

Chapter 3 - Containerization, Virtualisation and Separation of process/progams

The technological routes of virtualisation and containerisation, from the Mainframe to jails to containers.

Chapter 4 - Scaling by the majors

A description of Google, Microsoft, Yahoo and others scaled their own operations in the early 2000s.

Chapter 5 - Selling Scale

Heroku and then the birth of AWS.

Chapter 6 - Everyone in the Cloud

The growth of Cloud Computing with AWS, Azure, GCP, OCP, Hetzner, Digital Ocean etc, etc.

> Google just threw big numbers at it to make sure they took the biggest piece of the pie.

And, ya know, invented cgroups and everything.

If they were so innovative then why didn't they take the next step and make something like Docker?

Because Docker is also innovative?

What is your argument here? If Google doesn't innovate everything, it can't be innovative?

This is nonsense.

I was put off by your sarcastic dismissive "Ya know". I may have a controversial opinion, but I don't think it warrants that kind of flippancy.

I'm sorry - I'll try to be kinder. To explain myself, I too was put off by your dismissiveness of Googles own key contributions. But that doesn't mean I can't say that kindly.

For whatever it's worth, you have a very cogent way of making your point.

Thanks. I must admit I've learnt a lot about Google's contributions from this thread, so it's been productive overall.

I was never really aware of heroku.

I knew app engine and other things for much longer.

I would not subscribe to your interpretation.