This setup is very, very simple and scalable. There is very little to gain IMO on moving to Kubernetes.
Consul, VSphere and load balancers have APIs and you can write tools to do everything that K8s does.
In some networks DNS failover is really not that great, so at least a virtual ip needs to be used.
For people wondering why not use consul template, this has benefit of understanding haproxy, and it minimizes numbers of restarts to make changes.
Using haproxy over and has the benefit that if Winkle or consul goes down things continue to work, just not updates.
Also, don’t confuse 100 gbe networks where spine links are 100 but the node links are only bonded 10s (much more common at $fang).
It's a necessity really. All storage at Google has been remote and distributed for at least the past decade. That puts serious demands on network throughput if you want your CPUs to actually do work and not just sit there and wait for data.
Here's some detail as of 2012: https://storage.googleapis.com/pub-tools-public-publication-.... Note that host speed is 40Gbps. And here's FB talking about migrating from 40Gbps to 100Gbps in 2016: https://code.fb.com/data-center-engineering/introducing-back...
I know there are a lot of companies that try to put some lipstick on nfs pig and call it reliable/scalable/etc. so long their clients don’t actually try to run it at scale or don’t complain too publicly when they try and can’t, they are able to get away with it.
Of course that over longer networks it isn't suitable as the roundtrips have too much latency, other than that, is your experience much different regarding nfs?
Supporting all of a Fortune 500’s business operations is very reasonable to call ‘scale’ in the normal world.
Your comment is a like a billionaire claiming that somebody that managed to hit 30 million isn’t rich.
Many of the major infrastructure/platform vendors are rolling out their own distribution of Kubernetes either as a cloud service e.g AWS, Azure, GCP or on premise e.g. RedHat.
So I suspect they are going to try and differentiate on features and ease of use and make it as hard as possible to move anywhere else.
I work at Google on a large team of engineers dedicated to making it as easy as possible to use.
For a while, "AWS" and "the cloud" were practical synonyms; it was very rare that anyone meant not-AWS. In my opinion, Kubernetes is a major piece of Google's strategy to turn that tide and improve their marketshare.
Does throwing "a large team of engineers" at a problem typically result in something that's "as easy as possible to use"? "Design-by-committee" is not a term of endearment.
A sibling comment at https://news.ycombinator.com/item?id=18958077 notes that "there are a ton of nicer UIs for Kubernetes", but that they're sold separately as proprietary PaaS platforms. Even if we pretend like GCP/GKE isn't one of them, Kubernetes-The-Platform will be impacted by the interests of its primary vendors and advocates, whether or not certain teams at Google are keen to admit that.
One thing I've really appreciated, was how one could enable/disable things based on the binary version rolled in, and if it's rolled back the state goes back.
Basically something like this:
{
new_exp_feature = (binary_compiled_after_changelist( 123456789 ) || binary_compiled_with_cherrypicks( { 123456795, 1234567899 } )
}
With git/hg - you no longer have such monotonic order, but also that monotonic order worked best with monorepos (or maybe I'm just too narrow-sighted here)...
All of this seems way more complicated than the tools we use at my company. Is there a specialized need here I’m not seeing?
The evaluation rules are merely a borgcfg artifact.
Disclaimer: I maintain borgcfg.
https://github.com/ksonnet/kubecfg is an attempt to reboot the borgcfg experience with k8s + jsonnet
For example, {borg,kube}cfg allow you to import an existing config and override it so you can adapt it to another scenario (different things in different clusters, like prod vs staging, or a cluster has a new feature while another one doesn't etc).
Furthermore, the overrides are described with the same "shape" as the things they override, and can override things that weren't necessarily marked as overridable.
Compare this with the current state of affairs with helm, where the only way for users to be able to inject e.g. the resources requests and limits in a pod spec is for the original template author to have foreseen that need and explicitly added a hook in the template so that values from another file (the values.yaml) can be injected into it.
spec:
imagePullSecrets:
- name: {{ .Values.image.pullSecret }}
containers:
- name: {{ template "mycontainer.name" . }}
......
resources:
{{ toYaml .Values.resources | indent 12 }}
volumes:
......And what do you think is the primary driver of said cost of living?
I believe there is a ton of arbitrary complexity in the system, and though it's not specifically created, it definitely grows if it's not checked and entities with power have no reason to do that.
Google, Oracle, MS, Governments, Banks - have very little incentive to clear the weeds, usually just the opposite.
Wherever there is steady profit, there are layers of cruft.
Mostly NIMBY-driven refusal to build more housing and transportation infrastructure. It's not like the US is lacking for land. There's no reason a dilapidated teardown-ready shack should cost $2M, no matter where it is.
" There's no reason a dilapidated teardown-ready shack should cost $2M, no matter where it is."
The cost is not the shack, it's the land it's sitting on.
The higher the salaries in the valley, the more that dilapidated shack will cost.
No, it's just 'supply and demand' neither are 'high or low' necessarily.
SV has quite high wages, that's a huge driver of demand.
The residents of SV do not want to be like NY or Hong Kong, that is their choice. It's the choice many, many places make as well. Zurich, Paris, even London, they don't live in high rises.
The attractiveness of Cali in many ways is that it's not entirely flooded/urban like NYC.
There are not just arbitrary ways to create more homes, it has an effect on the situation.
There's still a learning curve, but it's much more humane than Kubernetes.
Kubernetes is very simple. And it will become much more complex with the growing hardware, network, and applications it's trying to manage.
What's missing is that there is a layer of complexity on top of k8s are still left for figuring out. And I think the operator's pattern is the right abstraction for service jobs. Some kind framework is still needed to handle the batch/offline workloads though.
There’s some work going on to have something more user-friendly (think Google’s Piccolo) - https://github.com/stripe/skycfg (disclaimer - I contributed to this project)
Have you tried Nomad?
actually I used kubeadm and the higher the version was going the better it worked for major upgrades.
At the moment with the new master upgrade methods I did not have any problems so far. on two clusters.
Sadly I created my cluster with an "external" etcd, beside that it is internal and also tried to maintain my own certificates, which is now a pita. (at the time cert handling wasn't as good in kubeadm as it is now).
Also I have a CloudConfig/Ignition Config creator which can bootstrap all necessary configs to bootstrap a kubeadm cluster on ContainerLinux/Flatcar Linux. So if I really have time I can just recreate a new cluster and move everything over. (I.e. the only thing which is problematic in "moving" over is the database created with kubedb)
Also you can use keepalived as your kubeadm load balancer.
Also I'd only go with a managed k8s solution and I'm not sure I'd consider k8s for older or non-microservice/containerized architectures. In the later case though I don't think there's anything better out there in terms of orchestration.
It would be really great if there is a human I could speak to at GitLab about this. I've put my email in my profile.
I suggest every startup use a hosted k8s solution, which takes care of most things like authentication, networking, monitoring, updating, etc.
also keep away from templating system such as jsonnet which is a huge overkill. you will end up writing a lot code you will hate to read later. instead write your own yaml builder in CI, together with parts that do docker image building, and code that deploys the microservices
imo Google did a really smart move with open sourcing k8s, as a latecomer of cloud provider. now infrastructure become so insignificant since everything runs on docker and pods.
Most importantly: I want a lot fewer moving parts than it currently has. Being "extensible" is a noble goal, but at some point cognitive overhead begins to dominate. Learn to say "no" to good ideas.
Unfortunately there's a lot of K8S configs and specific software already written, so people are unlikely to switch to something more manageable. Fortunately if complexity continues to proliferate, it may collapse under its own weight, leaving no option but to move somewhere else.