I don't think it was doomed from the start; Mesos was deployed at Twitter 4 years before k8s was even released. The "problem" with Mesos was that it was focused on a future that never panned out. The "killer app" for Mesos was Spark, and the problem they were focused on solving was resource allocation for batch jobs. Mesos was supposed to be a replacement for YARN. Even Marathon, which was the de facto application launcher, was pitched as a "meta-framework". Marathon wasn't for launching services, it was for launching custom frameworks. They never really pivoted from this, and the writing was on the wall when Mesosphere decided to make Marathon as proprietary as possible and fully integrate it within their commercial solution. Once Marathon was gone, Mesos didn't really compete with k8s anymore unless you wanted to write your own framework.
Mesos started as a research project at Berkeley in 2009 and was originally focused on cluster computing frameworks like Hadoop. From the paper: "We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI." It actually predates YARN by a few years. But, it very quickly (in 2010) saw production use at Twitter as the foundation for Twitter's custom PaaS which was later open sourced as Apache Aurora.
Marathon's main use case was actually for running microservice application in containers, which is why it has some advanced features around managing groups of containerized apps and their dependencies. The "meta-framework" use case for launching custom frameworks was also important but basically just needs Marathon to keep a container alive. Mesosphere never made Marathon proprietary. The full code is still OSS here: https://github.com/mesosphere/marathon/ Our commercial product DC/OS just added advanced workflows through a UI on top, and better integration with the rest of the components around Mesos.
Mesos was originally an academic project out of UC Berkeley. I'm not aware of what industry connections there were, but the story is not at all similar to Kubernetes and Borg.
Aurora was Twitter's contribution -- a bit missing piece of Mesos at the time. It definitely steered Mesos towards solving for Spark, but I'm really not sure what Mesos was actively solving before then.
Two concrete examples: Mesos maintenance mode vs Kubernetes' cordoning and eviction APIs, and Mesos's default behavior when a Node is suddenly powered off vs Kubernetes'.
What was Mesos default behavior when a node was powered off?
Kubernetes will, after a period of time, garbage collect the Node and schedule replacements as dictated by the controller.
Surprising because one of the driving forces behind accelerating adoption of on-demand IaaS and various PaaS like Serverless is that too many expensive server resources lay idle. According to James Hamilton, chief Data Center architect at AWS, server utilisation remains very low (10%–15%) despite servers being the most dominant cost of building and running a data center (which is to say, folks pay through their nose for servers yet those are under-utilized by a huge margin) [0]
Edit: sentence fixing
There was a period when it wasn't clear that you didn't need both resource management and container orchestration. One of my colleagues was quite convinced at the time that we needed both Mesos and Kubernetes. If course, the market coalesced around Kubernetes which largely backfilled the missing capabilities.
ASF is just a shelter for providing governance, infra, a legal framework for donations, and credibility to projects that went unprofitable, or weren't intended to be profitable to begin with, such as complements to commercial software. It's a responsible way to dump software you can't realistically maintain going forward, to give it a chance to build up a community by those depending on it, and often accompanied by an initial contribution to ASF. While not every software project is doing well for sure, whether ASF or not, I think the above sentence really misrepresents what ASF is doing (and arguably is doing with a lot of success).
For one, Kubernetes was, at least to some extent, a rewrite and extraction of functionality built at Google, from their production orchestration ecosystem that is Borg. The fact that Kubernetes was heavily influenced by a successful, large solution in this space allowed it to leapfrog, at least a bit, the competition.
Mesos was trying to become a solution seemingly from scratch. I worked with and interacted with a number of Mesos project members and committers, and while they were generally bright folks, their industry experience was surprisingly shallow. In my experience, Mesos changed frequently and there were enough unknown TBD components at the edges of the ecosystem to make it somewhat volatile.
Within a year Kubernetes was waaaaay ahead and Mesos started considering pivoting to support k8s.
I no longer see a purpose in Mesos, and frankly, that's okay. Too many Apache projects are on eternal life support, and they lower the bar for the rest of the Apache ecosystem, which has sort of begun to earn a poor reputation for quality (ala some of the fringe Boost libraries). Apache Foundation is no longer a label for solid, reliable software.