How do I install GKE on my servers? ;)
> By hand it’s not trivial to get a secure install.
The default install (basically, adding a repo and apt-get install docker-ce on Debian and derivatives - trivial to automate with Ansible) is reasonably secure if you view Docker as a tool for packaging and task scheduling with some nice extras and don't buy the marketed isolation properties. It only listens for commands on a local socket, and permissions are sane. I haven't looked into Swarm mode protocol traffic, though but I don't think it's tweakable anyway.
> The myth that kubernetes is complicated is largely perpetuated by the same kind of people who say React is complicated: the people who’ve not tried it.
I've tried K8s. I've set up a test cluster, it worked, I wrote some YAML, it worked, all good. So I've worsened the conditions (explicitly going into "I want things to break" territory) and made it fail. I've researched how hard it is to diagnose the problem and fix it - it happened to be complicated. At least, for me. Just felt that "if something goes wrong here, I'll have bad time trying to fix it". Surely, this is not the case on GKE where you don't run and don't manage the cluster.
I had somewhat similar experience with Docker and Docker Swarm mode, and it was significantly easier to dive into the code, find out the relevant parts and see what's going on.
> difficult with Swarm, like TLS termination
YMMV, but I just add some labels to the service and Traefik does the rest. ;)
(But, yeah, Traefik with Swarm requires some reasonable but not exactly obvious networking logic. May take one deployment of "why I'm getting 504s?!" to figure it out. And Traefik needs access to manager nodes to work with built-in Swarm service discovery.)
Great question! All major Cloud providers offer managed Kubernetes services:
https://cloud.google.com/kubernetes-engine
https://azure.microsoft.com/en-us/services/container-service
https://www.ibm.com/cloud/container-service
The choice is Cloud + Kubernetes vs. roll everything on your own hardware.
Running your own hardware is a major IT effort. Kubernetes is just a part of that effort. Then you have take care about planning, provisioning, logging, monitoring, alerting, auditing, networking, storage, oncall.
You either run Cloud + Kubernetes and get rolling in 15 minutes, or you hire IT headcount.
For Enterprise nerds VMWare has also just launched a Kubernetes service, so the #1 Enterprise VM supplier just added k8s on-prem for everyone who is in a hybrid cloud situation.
You can always rent dedicated hardware – much cheaper than renting similar amounts of virtual servers, and the same administration and operations costs. It’s often even significantly cheaper than what GCE offers – but of course, setting up Kubernetes is more complicated.
If you are a small startup looking to validate market fit, your best bet is Cloud + Kubernetes. If you are an established business with millions of daily customers and serious IT headcount budget, you may look into roll-your-own. The best orchestrator at that scale is, again, Kubernetes.
For a startup, building a datacenter isn’t required in the beginning.
I meant, literally renting existing hardware, in an existing datacenter. This is just a single step from renting VMs at AWS or GCE, but already improves costs significantly.
Edit: got user names wrong
I've deployed a handful of servers automatically myself this way, and I'm still a student.
Saving about an order of magnitude in terms of costs, gaining flexibility, and the only additional work is so low that it costs a compsci student half an hour once (aka ~10€ in wages) — there's no reason not to do it.
I agree with what you say. I'm not tying to say people should all jump to k8s. Having options on the market is great.
But I was trying to refute the notion that Kubernetes has no advantages unless you're running a huge cluster. My main points where:
* It works great with 1 node.
* It comes with many features that Swarm does not have that are useful even at 1 node (PersistentVolumes, StatefulSets are biggest for me, though there are _many_ more I wouldn't want to go without anymore).
* Docker is not trivial to set up, either.
> How do I install GKE on my servers? ;)
Yes, of course. I was just saying there's a solid option to start exploring quickly.
> It only listens for commands on a local socket.
This is kind of a non-starter, isn't it? Of course it's easy to apt-get install docker, but then you want to control it remotely, right? Once you realize how nice it is to control Docker remotely, it's hard to imagine life before.
However, Swarm is undeniably simpler to work with unless you have very specific requirements that only K8S provides. The yml file is incredibly simpler.
Docker Swarm is the Kotlin to Kubernetes' Java. It's a much pleasanter and much less intimidating way to build production systems that scale pretty well.
Kubernetes needs you to have load balancers setup which can talk K8S ingress. Bare metal setup for k8s is a huge effort (and I have stuck long enough on #sig-metal to know) as compared to Swarm.
You should try out Swarm - you might choose not to use it, but you will enjoy the experience.
Do you have a concrete example of what you ran into? What do you mean by "secure install"?
on osx its just a matter on installing docker4mac and other linux distributions has pre-made packages. I am a linux noob and I was able to setup a 20 machine cluster with swarm trivially on centos7.
I was acutally surprised that I was able to do that so trivially, given I have minimal linux admin experience if any.
https://docs.docker.com/engine/swarm/how-swarm-mode-works/pk...
You're absolutely correct. Kubernetes has its advantages, even in a single-node setup. What many others are pointing out is that it also has significant disadvantages, too.
> but then you want to control it remotely, right?
By the way, I do talk to Docker (Swarm mode or standalone) deployments remotely, forwarding the socket via SSH.
ssh -nNT -L /tmp/example.docker.sock:/run/docker.sock example.org
docker -H unix:///tmp/example.docker.sock info
But, really, there is Ansible for this. Declare `docker_service` and get it deployed (or updated) as a part of the playbook. And ELK (with logging.driver=gelf for the containers or application-level) for logging.
(BTW, what's nice about K8s is that IIRC it allows to exec into pods with just `kubectl exec`. With Swarm you need some script to discover the the right node and container ID, SSH into it and run `docker exec` there - no remote access.)
Please get some experience with that, then re-evaluate whether Docker or Kubernetes is easier for small deployments.
But with your easier Swarm setup, how do you then attach cloud disks directly to your docker container, like a PersistentVolume affords? That one feature makes basically anything worth it, IMO. Most apps are stateful.
Usually I just pin containers to hosts. This is fine on a small setup. In fact, many of my small setups are just a single host.
Try ProxMox. They have it built-in.
Yes, true, but that's apple vs oranges. Swarm doesn't compete against GKE, it competes against Kubeadm. Because sometimes one actually can't or won't use cloud services.
I don't understand ur comment, yea using something in preinstalled in the cloud is easier than installing something on your own.
$13 per month per egress rule, bandwidth costs a factor of 100 more expensive than dedicated hosters, and costs for hardware 10x of what dedicated hosters offer.
I’m not sure what your definition of "more expensive" is, but compared to renting dedicated hardware (e.g., from Hetzner), or colocating, GKE is significantly more expensive.
Of course, if you’re on the scale of Spotify, you can get much better deals – but from what I see on https://cloud.google.com/products/calculator/, it’s not cost-effective.
That’s why I disagreed with the "GKE isn’t more expensive".
depends on what scale your at and the fact that your vendor locked, can't use GKE in other data centers
I regularly use a script like this, and it's never failed me unless it runs out of disk space: https://github.com/coventry/InformationDropout/blob/master/p...
I haven't used but I have used DC/OS, Mesos, and Marathon extensively, which is not a setup I'd do for a small number of nodes personally.
But I stress it's not about the scaling. It's about the features even at a single node. I wouldn't be spending time writing this stuff had it not been revelatory for me.
PS. If you're not able to try it on GKE, there's also https://github.com/kubernetes/kops. GKE is great for trying, at least, though. Just get a cluster running and get rolling with the concepts. Then you'll know what it's all about. There isn't that much to it, and what there is to it is great and well documented with a great community.
I've personally tried both orchestration options and much prefer Swarm to smaller deployments. Sure k8s could prepare you better for the future if your app takes off to the moon but that's just premature optimization.
Also I challenge the $10 because what cloud instances plus ancillary services add up to $10? I’m talking 2 CPUs and 7.5gb RAM.
I would like to note that DC/OS now has Kubernetes [Beta]. You can manage Kubernetes resources on your DC/OS cluster now.
Here are the instructions on Github to get started: https://github.com/mesosphere/dcos-kubernetes-quickstart You can also join the Slack to help you on you out: http://chat.dcos.io/ (the #kubernetes channel)
Would you use DC/OS-marathon (vs k8s) now if you were to make that decision now.
I've heard that mesos stack is good for machine learning/big data stacks but how does marathon compare for deploying webapps.
Despite going with mesos I really had to contend with the fact that k8s just has way, way more developer support - there are so many rich applications in the k8s sphere. Meanwhile I can probably name all the well supported Mesos frameworks offhand. Next, marathon "feels" dead. They recently killed their UI interface as I imagine that they are having trouble giving resources to marathon. 3 years ago I wanted a reverse proxy solution that integrated with mesos as well as non-mesos services so I hacked Caddy to make that work [1]. 3 years later, I was looking for a similar solution and found traefik. It claimed to work with mesos/marathon, but the marathon integration was broken and the mesos integration required polling even though mesos had an events API, so I hacked traefik to make that work [2]. On the other side of the fence, you have companies like Buoyant who rewrote a core piece of their tech (Linkerd) just to support K8s (and only K8s). This has a compounding effect, where over the years things will just become more reliant on assuming you are running k8s.
That "cost" you pay to setup Mesos/k8s is usually a one time cost on the order of a month. I feel however, that k8s is going to give you a better ROI (Unless, you are managing 100s of nodes with Spark/YARN/HDFS, then Mesos continues to be the clear winner).
[1] https://github.com/mholt/caddy/pull/40 [1] https://github.com/containous/traefik/pull/2617
The DC/OS stack is nice for web apps, APIs, scheduled tasks, etc once you get it up and running. If I were just deploying small APIs and web apps, I would use something more managed like Lambda or Heroku/similar myself depending on the use case. That said, I'm more of an app dev than an ops person.
https://kubernetes.io/docs/tutorials/
(Disclosure: I run CNCF, which hosts Kubernetes and funded the EdX course.)
I’ve had all sort of difficulties installing Docker. By hand it’s not trivial to get a secure install. Docker machine is great except it’s often broken. The Docker machine dev team is a tired, understaffed bunch that’s always playing a sisyphean whack-a-mole against dozens of cloud providers and very needy posters on Github, myself included.
Kubernetes on the other hand is trivial with GKE. It’s great for single node deployments. I run a single node on GKE and it’s awesome, easy, and very cheap. You can even run preemptible instances. The myth that kubernetes is complicated is largely perpetuated by the same kind of people who say React is complicated: the people who’ve not tried it.
And like React, once you try kubernetes you never go back. Kubernetes is actually the orchestration equivalent of React. You declare what should be true, and Kubernetes takes care of the rest.
And the features it provides are useful for any-sized application! If you try kubernetes you quickly discover persistent volumes and statefulsets, which take away most of the complexities out of stateful applications (ie most applications). You also discover ingress resources and controllers, which make trivial so many things that are difficult with Swarm, like TLS termination. Swam doesn’t have such features, which any non-trivial app (say, Django, wordpress, etc) benefits from tremendously.