The thing is that using a cloud as a glorified hypervisor for home-baked VMs doesn’t work economically. Colo or even running a DC is cheaper. Cloud generally only makes financial sense if you are using the managed services. And of course the instant you do that you sacrifice portability to a greater or lesser extent. This is of course deliberate on the part of the cloud providers.
It makes sense to pick 2 clouds and go all-in on them using their native and managed services to the maximum. You will need to maintain two parallel skillsets to do this. Completely forget about any layer that promises to abstract it, they are all red herrings if not outright snake oil. Whether that’s IBM consultants, or Terraform.
I don't want to use them, but it looks like there might not be any alternatives for only some very few hours on such a cluster.
Moving data from one provider to another will quickly drive your costs up.
OpenShift isn't any different. And in fact Docker invested a ton of time into an entire product, built around Terraform, that manages ecosystem deployment to any cloud that a lot of people wanted but couldn't buy. In fact that was probably a more sellable product than trying to shove Enterprise Engine and Docker Trusted Repository down people's throats.
If you boil it down the common pattern seems to be: cloud lift and shift of legacy VM from on-prem DC is more expensive. The lift you get from cloud is PaaS and the SaaS that's included to manage it. But very few are cloud native ready and take the perspective that: we'll lift and shift it today and then our next project is cloud native transformation. Yeah... That latter part rarely happens. And AWS, GCP and Azure all profit.
In one breath we hear that only three software vendors (Google, Amazon, Microsoft .. maybe DO?) have software worth consuming, and all other software vendors that ship this stuff “abstracts” them is snake oil... MongoDB Enterprise... Confluent Kafka... Elastic Cloud Enteprise... Pivotal CF, Red Hat Openshift, Hashicorp Nomad / Terraform / Vault etc - all of these provide valuable software that runs on any cloud, and often has a multi cloud control plane (that’s increasingly Kubernetes based).
let’s never use any of that, and screw the whole software industry for the cloud vendors because their stuff isn’t just a proprietary veneer of automation around those products charged by the hour?
on another breath we are told that the cloud providers charge too much for their VMs. And we think they’re not charging too much for their proprietary services?
Firstly, Most “managed cloud services” are not “managed” in the traditional sense. They’re hosted, no different from Dreamhost or a bazillion other hosted offerings, often with similar tradeoffs. It’s a testament to cloud marketing that people believe there is something magical about Amazon RDS for your Postgres instance. It’s a nice automated setup of volume replicated active/passive Pgsql.
There are many, many other ways to do this with open source or proprietary software with varying degrees of automation - maybe you don’t care, that’s fine, but I’m not sure delta between running in a DC/colo vs the EC2 costs is worth it for some proprietary software bits by the cloud vendors. It’s not magic, it’s just software.
Similarly to say all abstractions are snake oil is fashionable but also hypocritical. Kubernetes is on fire lately because it is an abstraction for your work loads, a universal control plane, and universal cloud API. Is that snake oil? Serverless (the framework) makes developing on Lambda or other FaaS’ sane - is it too snake oil? Heroku or Cloud foundry lets you push your apps and not worry about the plumbing on EC2 or the cloud of your choice (even on a colo/DC!)
But most importantly: You’re not locked into lowest common denominator (what does that even mean?) with any of this - you can use any proprietary cloud service you want...the stuff you don’t care about - the VMs, network and storage - is the stuff abstracted (and usually all the proprietary knobs like Azure advanced networking or Google metadata/DNS are all available).
Where is the problem? Are Elastic Beanstalk or Google App Engine really superior economically and functionality wise?
Terraform is not about the different codebases, it’s about glue code in a standard language to assemble all these cloud services. Crossplane.io is trying to do this via Kubernetes CRDs. Would you rather use Cloud Formation and JSON, really?? I’ve seen some monster CF scripts - they’re hard to maintain and debug compared to TF.
Let me give you a trivial example: GCP gives you a lot of flexibility with CPU and memory when creating a VM. Let’s say your workload is ideally suited to some weird combo, like 5 cores and 13G or something. On GCP that’s what you provision and that’s what you pay for. AWS and Azure offer fixed sizes, so you have to round up to 8 and 16 (and pay for it). So if you want to be cloud-agnostic there’s one cool but very basic feature you just can’t use.
Once you start digging into this stuff this keeps coming up: something that’s efficient (cheap) in one but not in another means: do I do it the same and pay more, or do I diverge and accept that I’ve got 2 configurations for this feature now, and save some money. Multiply this by 1000 special cases and then you find that trying to make one size fit all is a wild goose chase.
All Terraform really offers is doing this in similar syntax for the subset of each cloud’s features that Terraform knows about, and extending Terraform yourself for anything it doesn’t. Yes, I would rather use the native thing for each cloud, even CloudFormation. ARM Templates and DSC are actually quite nice once you get used to them!
It's interesting, I use BOSH + Terraform all the time, wherein BOSH exposes GCP's CPU/memory flexibility, the various Azure NIC/LB/availability set options, AWS' different disk types, etc. Those differences can be modularized, so that 95% of your configuration templates are identical across clouds, and the last 5% maps to specifics.
I'm sure it's not all that different from Terraform w/ modules, though my main problem with Terraform is that it doesn't constrain you into a "do the right thing" path, it's too easy to create a mess.
Anyway, IMO these kinds of differences really aren't hard to handle and it's valid to prioritize cloud-independent configuration if that's what you want/need. It allows the main configuration and installable software to be cloud-independent, dramatically easing testing. We're seeing this drive with the flocking to Kubernetes which enables cloud-independent networking, storage, and compute.
I think differing opinions on this are normal/fine, but i have to wonder why the single-cloud proponents use words like "snake oil" as if to completely discredit a different set of priorities.
Yes, every time. The "multi cloud" support in TF is just marketing, doesn't work. CloudFormation, no space, btw.
YES! I might be the only one that wants this, but this is the future I want[0][1][2]. As technology makes it easier and easier to run a cloud compute/service provider, more competition will improve the offerings.
I don't know what "shit tier providers" you're referring to, but I'm not convinced of the additional value AWS is delivering when running certain services (I'm thinking postgres) which are already pretty excellent by themselves. In the end you need to hire for AWS expertise (and spend time boning up anyway/reading docs when things go wrong), and for 90% of apps 1 mid-tier postgres instance (not even a highly-available) instance is all you need until you really get traction and can afford to hire someone to look after it. I think the future is going to see the bifurcation of management (making sure it's up/efficient) and hosting (providing hardware), and plummeting prices for both as more people enter the market.
The super perplexing thing is that cloud providers jumped on the kubernetes bandwagon so quickly and this is what made this commoditization possible. Pessimistically speaking, this means either they're either about to butcher the cross-cloud usability of kubernetes (i.e. drift between EKS,AKS,etc) or kubernetes was a wolf in sheep's clothing from the start.
I've been cautiously watching the Service Catalog[3] feature in Kubernetes because I think it's one of the places where vendors are exerting the most control to see how it goes -- I've brought it up before, but the "Service Catalog" is basically identical to the user Operator (Controller + Custom Resource Definition) pattern, but for some reason it's being treated as different...
[0]: https://news.ycombinator.com/item?id=19170073
[1]: https://news.ycombinator.com/item?id=18121150
[2]: https://news.ycombinator.com/item?id=18015057
[3]: https://kubernetes.io/docs/concepts/extend-kubernetes/servic...
My view remains that this was the purpose of Kubernetes: to scorch the earth ahead of ECS. Google had nothing to lose by doing so.
> I've brought it up before, but the "Service Catalog" is basically identical to the user Operator (Controller + Custom Resource Definition) pattern, but for some reason it's being treated as different...
The Service Catalog is based on the Open Service Broker API, which in turn was extracted from the Cloud Foundry Service Broker API. I hear folks say "it's not declarative", but typically this turns out to mean "it's not a YAML file". The fact is that OSBAPI is literally about telling the platform: hey, I need a service for this app. The brokerage process then works out how that's achieved. One such mechanism is Operators, there are also brokers that can apply Helm charts.
The major difference is really about the CRDs as the point of interface. There's a fork or project somewhere which presents OSBAPI as CRDs (ie, YAML submitted to a remote REST endpoint) instead of as CLI commands (ie, JSON submitted to a remote REST endpoint). I feel like that's the best of both worlds. You keep the internal consistency of service discovery, service binding and service injection, plus the thing that really adds value: checking the YAML into revision control.
Disclosure: I work for Pivotal. We compete with Red Hat / IBM. We also cooperate with them on OSBAPI and a number of other projects.
I agree, but avoided saying it to try to not seem like a crackpot. I'm fairly sure I have a comment to the same tune somewhere in the back of HN somewhere (or maybe somewhere else).
Also in the sounds-like-what-a-crackpot-would-say category of thoughts -- I don't trust the CNCF. Similar to how I don't trust the OIN.
> The Service Catalog is based on the Open Service Broker API, which in turn was extracted from the Cloud Foundry Service Broker API. I hear folks say "it's not declarative", but typically this turns out to mean "it's not a YAML file". The fact is that OSBAPI is literally about telling the platform: hey, I need a service for this app. The brokerage process then works out how that's achieved. One such mechanism is Operators, there are also brokers that can apply Helm charts.
No comment about this, "declarative" is just tacked on onto things as marketing fluff these days. OSBAPI is plenty declarative to me; you're telling it what you want instead of how to provide what you want.
> The major difference is really about the CRDs as the point of interface. There's a fork or project somewhere which presents OSBAPI as CRDs (ie, YAML submitted to a remote REST endpoint) instead of as CLI commands (ie, JSON submitted to a remote REST endpoint). I feel like that's the best of both worlds. You keep the internal consistency of service discovery, service binding and service injection, plus the thing that really adds value: checking the YAML into revision control.
Though heavily simplified, do I have the difference (written below) correct?
API version:
0) User wants cloud provided service X
1) User makes service catalog CLI request (a web request is made to the broker)
CRD version:
0) User wants cloud provided service X
1) User creates a CRD
2) Controller sees created CRD and makes web request to the broker
If this is right, it seems like a very small difference.
To be fair, OSBAPI is a bigger upfront effort for a service author than fiddling with kubebuilder and sorta-kinda doing some basic service-y stuff yourself. Plus CRDs are the cool thing right now and I expect they will continue to build in momentum.
Where it will come back to bite everyone is that every service will have its own Operator and its own CRDs and these will often be different in shape, behaviour and conventions than other Operators and CRDs. It'll be like the bad old days of every piece of software shipping with its own pile of bash scripts for management and twenty different process managers. You know: the sort of thing Red Hat helped to bring an end to.
After we reach the point where you're spinning up whole clusters because the PostgreSQL operator doesn't play nice with the Kafka operator on Tuesdays when the breeze is too strong, someone wise and brave and famous on twitter will yell "this is insanity! Surely we can rationalise this into a single API, a single mechanism!"
And then OSBAPI will be reinvented (but ... declarative!) while I stand at a safe distance, teeth audibly grinding down to bone.
> To be fair, OSBAPI is a bigger upfront effort for a service author than fiddling with kubebuilder and sorta-kinda doing some basic service-y stuff yourself. Plus CRDs are the cool thing right now and I expect they will continue to build in momentum.
kubebuilder and the whole "ecosystem" for building controllers is unnecessarily hard to use and ideally shouldn't have been this way in the first place. Writing an operator is more painful than it should be, and I think many of the engineering decisions early on are what made it this way.
IMO CRDs are actually the defining feature of k8s but no one realized it until later. They set out to build a cathedral when they should have been building a more orderly bazaar.
> Where it will come back to bite everyone is that every service will have its own Operator and its own CRDs and these will often be different in shape, behaviour and conventions than other Operators and CRDs. It'll be like the bad old days of every piece of software shipping with its own pile of bash scripts for management and twenty different process managers. You know: the sort of thing Red Hat helped to bring an end to.
Right, but maybe this time people will solve that problem the right way -- by standardizing on the representation (CRD) in this case. I mean this in the general sense -- the case I think of is trying to capture all the different ways you can containerize something as they evolved over the years:
- LXC exists (let's call this an `ubuntu.com/Container` entity)
- Docker is introduced, people go wild (let's call this a `docker.com/Container` entity)
- OCI container spec gets written (Let's call this an `opencontainers.org/Container` entity)
- People want to just be able to make a "Container" that runs on whatever operator is available ==> They make a `foo.com/Container` that represents a the lowest common denominator
This kind of progression is the only way you can keep your rapid iteration but have consistent behavior -- foo.com takes the bullet to provide and maintain consistent behavior while the other container types can iterate as fast as they desire.
> After we reach the point where you're spinning up whole clusters because the PostgreSQL operator doesn't play nice with the Kafka operator on Tuesdays when the breeze is too strong, someone wise and brave and famous on twitter will yell "this is insanity! Surely we can rationalise this into a single API, a single mechanism!"
This I think is a symptom is an orchestrator/operators that aren't robust enough -- operators should not be able to take out other operators, except through very well defined integration points. Hard to write the interface in practice but I think we can do better than what kubernetes currently is.
I've been sitting on the concept for a kubernetes competitor that (I think) is simpler but haven't gotten around to writing it. Here are the broad strokes:
- The only pattern is provider/resource (essentially what k8s knows as CRDs and operators)
- Complexity increases only by composition (i.e. you want to write a `KafkaProvider`? looks like you'll need a `ContainerProvider`, `NetworkProvider`, etc)
- No GRPC (good 'ol HTTP1.1, upgradable to 2/3 in the standard web-compatible ways, with "Content-Type" if you really feel like you need to squash your request/responses, and SSE if you need to stream stuff)
- jsonschema + hyperschema + JSON-LD for automated everything (verification, request building, semantics parsing)
- Rust
- Single binary deployment (as in get your VM, maybe set some kernel flags, put a single binary on it, run that binary, and your cluster is up).
I've got pages of notes on what this could be but at this point it's vaporware -- Who knows if I'll ever get to work on this idea but I sure wish someone would. I don't think there are any companies willing (essentially, dumb enough) to try to compete with Kubernetes so I don't think anyone is even considering trying to build a new but refined k8s.
I think that your ideas have promise, but I caution that container schedulers a gigantic suckhole of tedious details. Pivotal and IBM wrote Diego and it's slightly older than Kubernetes. It's battle-tested but holy shit the zany one-in-a-million evil fluke bugs of doom happen every day simply because there are enough containers being run worldwide to hit them.
The "performance" bit is bandied about a lot, but I'm convinced that you can get far enough with `Content-Type` and bin-packed messages when you need them. You can request/receive binary with HTTP 1.1, that's how images and what not get to you -- if what I'm imagining is even relatively easy to implement, HTTP 1.1 would be strictly better than GRPC because of the flexibility (at the cost of sending a few more headers). Once HTTP 2/3 settle and see more adoption, GRPC's performance benefits would be reduced even further.
I'm not convinced I need to buy into all of GRPC to get efficiency from bin-packed messages across the wire, or bidirectional communication.
Is it perplexing? Didn't Google do the same thing with Chrome and Android? Understandably those aren't enterprise systems, but they successfully entered a market by open sourcing something that became wildly popular. In the case of Android and k8s, user cost is also reduced. Once it was clear that k8s won over mesos and docker's solution, AWS and MSFT didn't have much of a choice but to adopt it.
It makes sense for Microsoft/IBM/Oracle to jump on the k8s bandwagon because they're basically doing the worst/newest entrants but why AWS? They already had CloudFormation, along with lots of energy already dumped into SDKs for various tools (ECS, beanstalk, etc).
There are a few other technologies that could have done this before k8s really took off, like pre-k8s openstack -- why didn't that get adopted?
If they had chosen to use their own proprietary platform they risk Google and MSFT peeling customers away who want to work on a well designed open and portable orchestration platform. That might start slow but Amazon didn't get this big by allowing competition to take customers without throwing their own punches. They could have made their own _open source_ platform that would directly compete against k8s but by the time it was clear that demand was huge (I'm thinking early 2016) k8s already had to ton of momentum. Again they probably saw that and figured this was the best move with a high probability of success, even if it meant less lock-in.
They also have a whole slew of other products that serve almost exclusively as lock-in mechanisms. One truly portable product isn't going to be a huge deal.
EDIT: Distinguish proprietary and open source options in the second paragraph.
They have chosen that, but for rather than instead of k8s; they've announced that they are working on a fully managed system for k8s (Fargate for EKS, parallel to their existing ECS Fargate offering.)
I think it makes sense for AWS to jump on the bandwagon because they understand that Google is trying to commodify its complements [1]. The thing that AWS is betting on, I think, is not only surviving but thriving in a commodified environment [2]. The bet that AWS is making is that just like Amazon's retail business, commodification will hurt their competitors more than it hurts them, which will then allow them to make the most of the newly cleared competitive environment once the dust settles. Google's play, on the other hand, is to use the fact that Google is massively profitable elsewhere to allow Google cloud to run at negative profitability for a while in order to try to take a chunk out of AWS. The big loser I see from Kubernetes is Azure. Unlike Google, Microsoft does not have massive and growing profits from elsewhere. Enterprise and consumer Windows sales, while profitable, are on a declining profitability trajectory, according to Microsoft's own estimates. At the same time, Azure, in my estimation, doesn't have to survive extremely low margin environments in the same way that AWS can.
[1]: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
AWS doesn't act like they can win without obvious lock-in, they act like they can win by extending open technologies with proprietary management layers that customers will depends on and be locked in by, which they do in pretty much every area.
The VM providers where you purchase a years worth of VM time but aren't entirely convinced they'll still exist as a business in 12 months. There is a whole ecosystem of sketchy oversold VMs that you can't push to 100% 24/7 but are dirt cheap.
Not useful for business in itself, but 20 glued together...maybe.
That said, somewhat orthogonal to your point -- the amount of capital applied to create these gigantic platforms is not indicative of what people can and do create with internalized cost -- the F/OSS movement has proven that. People have created immensely valuable software, and internalized/shared the burden of creating that software. I also think that it is entities with large capital that strive (and almost always end up creating) gigantic platforms -- it justifies their heft/consulting teams, etc. motivated single contributors/small groups never set out to giant platforms, they usually build things that do one thing well (see unix) and compose (see unix) to do greater things.
More towards your point directly, you're right, a lot of things would stop making sense in a perfectly competitive market in the theoretical economics sense. Outside the fact that the model is wrong/incomplete (as all non-perfect knowledge models have to be), I do think this would lead to a certain number of competitors, but likely more than 0. My instinct is that the number of firms selling the product would be equal to the aggregated demand / costs to run a business selling the product -- and that ratio is probably greater than 1. Purely a spherical cow scenario but at least that's where I think it would go.
The only version I can see is one that runs on generic nix. i.e. Spin up a VM on each cloud (and in house) and run a stack on that which software glues it together. Redundancy of sorts.
...however if that glue actually works I might as well do that with 20 sht tier providers and let the redundancy cover gloss over it. No competitive advantage for cloud providers - and each of the cloud's various competitive advantages are by definition not hybrid-able easily.
Scaling? You're now dealing with multiple cloud's worth of different approaches to scaling. Good luck scaling that hybrid style in a resilient way.
...and all of this is just VMs. Add the 50 other offerings the average cloud has all with different scaling, quirks and the hybrid dream is deader than dead.