News for hackers. Stuff that matters.
Re:Four years after its release, Kubernetes has come a long way by tnolet on Wednesday June 06, 2018 @04:32PM (#17250848)
The only part missing in most Kubernetes write ups is that you probably don't need it. I'm hopeful most engineers recognise this, although my day to day experience paints a darker picture. I know of at least one hot startup deploying Kubernetes just to attract engineering talent.
Re:Four years after its release, Kubernetes has come a long way by indescions_2018 on Wednesday June 06, 2018 @05:23PM (#17251245)
"Kubernetes" is an Ancient Greek word meaning "more containers than customers"

https://twitter.com/srbaker/status/1002286820078571532

Re:Four years after its release, Kubernetes has come a long way by smnrchrds on Wednesday June 06, 2018 @07:30PM (#17252031)
I assume it is written in the style of "Ubuntu is an ancient african word, meaning 'I can't configure Debian."
Re:Four years after its release, Kubernetes has come a long way by StavrosK on Wednesday June 06, 2018 @08:38PM (#17252342)
It is, "kubernetes" is actually the root of the word "governor", with similar meaning in ancient and modern Greek.

The naming of the software makes for very unfortunate conversations if you're talking to other Greeks about it.

Re:Four years after its release, Kubernetes has come a long way by kbenson on Wednesday June 06, 2018 @08:10PM (#17252222)
This totally could have come from DevOps Borat. [1]

1: https://mobile.twitter.com/devops_borat

Re:Four years after its release, Kubernetes has come a long way by slow_donkey on Wednesday June 06, 2018 @04:49PM (#17250991)
What if you're on a hosted service like gke/aks? The work needed to maintain the system has been minimal for us with a lot of positives
Re:Four years after its release, Kubernetes has come a long way by bauerd on Wednesday June 06, 2018 @05:00PM (#17251082)
Yeah +1. I was super skeptical 1,5 months ago when I started my new job on a 22 node GKE cluster. "Surely K8s is overkill", "could've just used Nomad + Consul + Vault + X", "why not just use Fargate", "K8s seems like a distributed systemd", etc pp. Long story short, K8s on GKE just works, saves us tons of time, makes service-oriented architectures/microservices easy as pie and has led to a way better service level than the previous setup (Chef deployed on baremetal).

Yes, it probably is overkill for your 10req/s app, granted. But I highly advise everyone to at least give it a try out of curiosity, because a lot of hard problems at medium scale and above just go poof with K8s.

Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @05:08PM (#17251145)
> because a lot of hard problems at medium scale and above just go poof with K8s.

No, they don't? I don't know why anybody would just assume that something as complex as kubernetes would just run flawlessly once you actually try to run it on thousands of servers. Must be something to do with google PR because people definitely don't seem to assume the same for e.g. hadoop or openstack. Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running?

Re:Four years after its release, Kubernetes has come a long way by bauerd on Wednesday June 06, 2018 @05:20PM (#17251214)
I was commenting on a specific usage context:

>when I started my new job on a 22 node GKE cluster

>problems at medium scale

vs

>Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running

K8s obviously is not a silver bullet and of course there's ops work to be done. I can't comment on whether operating K8s clusters in other contexts makes economic sense, but I know it does for our current team.

Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @05:26PM (#17251261)
> makes economic sense [...] for our current team.

But the reason for that is not that it makes "hard problems go poof at scale". The reason is that you're using a hosted service where somebody else (in this case, Google themselves) takes care of the problems for you for a fee and - at small scale - you only have to pay them a fraction of a single operation's engineers salary for it.

So of course it makes economic sense for you to use a hosted service where the sharing economy kicks in, but your recommendation to use kubernetes because it solves hard technical problems at medium scale does not follow from that.

Re:Four years after its release, Kubernetes has come a long way by seangrogg on Wednesday June 06, 2018 @07:34PM (#17252052)
> [Google themselves] takes care of the problems for you for a fee...

GKE's services are, as far as I can tell from their pricing page[0], free. The compute running on top of GKE is charged at the GCE rate, the master node spun up by GKE is free.

Disclaimer: I work at Google Cloud, but nowhere near these offerings.

[0] https://cloud.google.com/kubernetes-engine/pricing

Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @07:48PM (#17252119)
How does this relate to my point that you're transitively paying somebody else to do ops for you? Maybe google's pricing model rolls this into the normal VM price? Or maybe it's currently offered at a loss to gain traction? Or are you saying they are not paying the SREs anymore and they work for free now?
Re:Four years after its release, Kubernetes has come a long way by seangrogg on Wednesday June 06, 2018 @09:44PM (#17252599)
Per their pricing, GCP doesn't charge for the master node VM that does the orchestration - you just get charged the normal price for VMs as if you'd provisioned them yourself. Thus, given the price of GKE is $0, the only thing I could see you "transitively paying somebody else" is experience - Google engineers become more versed in running managed Kubernetes and you don't. If the fee you perceive is dependency I'd agree - but I'd also opine that many startups/SMBs would be willing to accept that fee rather than onboard engineers to solve it.

As to why they're taking the loss on the master node VM, I don't know. I had previously expected that it was a cost and was quite frankly pleasantly surprised that it wasn't - it seems like the most obvious sell. If I had to guess as to why it's not my best assumption would be that there's far more to be gained in getting companies comfortable with scaling and from angles that go beyond just the strict monetary benefit of them going from 3 compute instances to 300.

Re:Four years after its release, Kubernetes has come a long way by jjeaff on Wednesday June 06, 2018 @06:47PM (#17251762)
If you are running on Google computer anyway, then I don't think you pay anything extra for gke. As far as I can tell, you are only paying for the VMs that you use. So the price for hosted k8s is already baked in whether you use it or not.
Re:Four years after its release, Kubernetes has come a long way by wmf on Wednesday June 06, 2018 @11:27PM (#17253024)
Yes, when you use GKE Google is operating K8s for you. Now ask yourself why Google and AWS and Azure offer K8s but they don't offer Swarm, Mesos, Nomad, etc. Maybe K8s solves some problems after all.
Re:Four years after its release, Kubernetes has come a long way by dominotw on Wednesday June 06, 2018 @05:41PM (#17251338)
It was a response to what if gke question. So it follows in that context .
Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @07:51PM (#17252136)
No it doesn't. You can not extrapolate from the fact that the hosted version "just works" that kubernetes at scale would also "just work" (kubernetes being the open source product that you run yourself here). Especially if the hosted version is offered by google and everybody knows that they employ truckloads of very good engineers keeping their stuff running.
Re:Four years after its release, Kubernetes has come a long way by brazzledazzle on Wednesday June 06, 2018 @08:13PM (#17252234)
The original comment asked “what about hosted kubernetes?” and the reply was discussing the benefits of that hosted solution. They didn’t say that running your own cluster would be the same. They didn’t even imply it.

Even if they had, it doesn’t justify your aggression. Maybe take a break from the internet for a bit to clear your head.

Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @08:35PM (#17252331)
Well I am sorry if you felt the wording of my comments was too "aggressive"; personal angle is weird though. Also I think you are trying pretty hard to misunderstand what I am trying to say. Obviously, using google's hosted version will make all your problems go "poof" at any scale, because they are Google's problems now. But that is precisely the point I am trying to make: It's not a feature of using kubernetes - it's a feature of paying Google!
Re:Four years after its release, Kubernetes has come a long way by mentat on Wednesday June 06, 2018 @10:01PM (#17252681)
Or Azure. Or AWS. Or probably in the future pretty much anyone. If you want to operate your own cluster you will be able to, for everyone else it will come from their cloud provider as a "cost of getting your business." This is a great thing.
Re:Four years after its release, Kubernetes has come a long way by brazzledazzle on Thursday June 07, 2018 @04:02PM (#17259678)
Yes, and that was their point: That the hosted solution did that. They didn’t say it was a feature of kubernetes. The entire thread started with a question about the hosted solutions and they talked about their experiences with that. Can you specifically point out where they say the hosted version’s benefit should translate to the self-hosted version?
Re:Four years after its release, Kubernetes has come a long way by akhilcacharya on Wednesday June 06, 2018 @05:58PM (#17251447)
Would it be accurate to describe some of the K8S ecosystem as building out equivalent services (monitoring, failover, health checks) to the cloud providers on top of a infra-agnostic layer?
Re:Four years after its release, Kubernetes has come a long way by pests on Wednesday June 06, 2018 @06:48PM (#17251773)
This is how I use it. My applications can now be cloud-agnostic besides any other services I use with that provider.
Re:Four years after its release, Kubernetes has come a long way by claytonjy on Wednesday June 06, 2018 @06:07PM (#17251499)
I'm another happy user of this approach. While I generally agree with other posters all over these comments saying it isn't worth it for small or even medium deployments, I'm happy using it with a team of 2-3 for a single client.

Much different use-case than most, though: automated machine learning pipelines via Pachyderm [1]. Would never have touched K8's if it wasn't for Pachd, and between their layer and GKE it's been quite painless. May have given up on Pachyderm if it wasn't for GKE; we're not cut out for maintaining K8's, and our first shot on AWS (pre-AKS) was not pretty.

tl;dr: K8's are great when it's all but completely abstracted away from you

[1] http://www.pachyderm.io/

Re:Four years after its release, Kubernetes has come a long way by bruinjoe on Wednesday June 06, 2018 @04:54PM (#17251029)
I see a lot of value in Kubernetes. For one thing I don't want to wake up at 1 AM if one of my node goes down and brings down with it important containers. If I scaled the environment with excess capacity then Kubernetes should start up new pods on the remaining nodes.
Re:Four years after its release, Kubernetes has come a long way by tnolet on Wednesday June 06, 2018 @04:58PM (#17251064)
That is an awesome feature of K8s and similar orchestators. It is however just an HA (high availabilty) problem already solved many times over in many ways. Almost every PaaS system has it. All cloud providers have it. Getting K8s (or any of its competitors) for HA is total overkill.
Re:Four years after its release, Kubernetes has come a long way by manigandham on Wednesday June 06, 2018 @06:58PM (#17251842)
We use a GKE cluster with dozens of preemptible/spot instances. Great savings for no extra management fee, and with a nice standard kubectl interface for deployments and changes.

VMs are the most reliable thing of any cloud, and often more reliable then their own managed services because of complexity. GKE does live-migration of VMs and Azure/AWS are close to the same thing, so building on that foundation and doing your own "managed" layer via Kubernetes provides a better experience, while also giving you that spot instance discount that you wouldn't get otherwise with PaaS.

Re:Four years after its release, Kubernetes has come a long way by kuschku on Wednesday June 06, 2018 @05:05PM (#17251121)
The point is that I can rent the cheapest $10/month servers I can find, a bunch of them, throw kubernetes on that, and it’ll still be up.

Same reliability as AWS, similar performance, much cheaper.

Re:Four years after its release, Kubernetes has come a long way by yongjik on Wednesday June 06, 2018 @06:04PM (#17251481)
Unless you're renting from multiple $10/month services, your service can be only as reliable as your service provider, which is probably less than AWS.
Re:Four years after its release, Kubernetes has come a long way by rhizome on Wednesday June 06, 2018 @09:48PM (#17252610)
Doesn't the same condition apply to AWS, only they're called "availability zones?"
Re:Four years after its release, Kubernetes has come a long way by bruinjoe on Wednesday June 06, 2018 @06:59PM (#17251850)
You hit it on the head. Weave Net is participating in the Istio project. I hope they can bring WeaveNet to Istio. That might be able to provide a multi cloud vendor service mesh.
Re:Four years after its release, Kubernetes has come a long way by kuschku on Wednesday June 06, 2018 @07:03PM (#17251875)
But that’s easily possible – you can use all the French hosters at once. Latency between them is practically nil because their data centers are only a few hundred meters away from each other.

Sure, in case all of Paris goes down, you’re down, too. But in day-to-day usage you can have good reliability on a shoestring budget.

Re:Four years after its release, Kubernetes has come a long way by int0x80 on Wednesday June 06, 2018 @06:58PM (#17251838)
What good $10/month servers do you know?

Genuinely interested.

Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @05:21PM (#17251229)
Nobody wants to wake up at 1AM because their singly-homed service just went down. Kubernetes might be a fine tool to achieve that, but I want to point out that there are much simpler failover solutions available. Failover is something you should definitely have, but also something you definitely don't need kubernetes for.
Re:Four years after its release, Kubernetes has come a long way by zzzcpan on Wednesday June 06, 2018 @05:31PM (#17251283)
In fact, kubernetes won't solve high-availability for you. It is completely orthogonal to failover.

EDIT: No, really, k8s won't make a typical RDBMS suddenly able to run in three datacenters across three continents.

Re:Four years after its release, Kubernetes has come a long way by pests on Wednesday June 06, 2018 @06:52PM (#17251793)
But it can. You can develop a custom controller that does manage a typical RDBMS to be able to run in three datacenters across three continents.
Re:Four years after its release, Kubernetes has come a long way by pritambaral on Wednesday June 06, 2018 @09:41PM (#17252591)
> You can develop a custom controller that does manage a typical RDBMS to be able to run in three datacenters across three continents.

Honestly, at that point, what does k8s get you?

AIUI, Kubernetes is fine for stateless systems, but it is really no better on the storing-state question.

Re:Four years after its release, Kubernetes has come a long way by wmf on Wednesday June 06, 2018 @11:33PM (#17253044)
It's easier to install Stolon under K8s than by hand. :-)
Re:Four years after its release, Kubernetes has come a long way by pests on Thursday June 07, 2018 @01:27AM (#17253497)
Consistency.
Re:Four years after its release, Kubernetes has come a long way by zzzcpan on Thursday June 07, 2018 @11:36AM (#17256747)
It was a trick sentence, RDBMSs that can do that and remain useful enough for at least some applications don't exist.
Re:Four years after its release, Kubernetes has come a long way by shaklee3 on Wednesday June 06, 2018 @09:35PM (#17252570)
Huh? One of the core features is making sure you have N pods running at all times. If one fails, it starts another.
Re:Four years after its release, Kubernetes has come a long way by zzzcpan on Thursday June 07, 2018 @11:32AM (#17256703)
This feature won't even work for anything designed for fault tolerance. I believe it can only work for things that either rely on other properly fault tolerant services or that are completely stateless, idempotent, so they can be retried and eventually consistent (in which case the state still has to be handled by properly fault tolerant systems). Either way kubernetes cannot help here.
Re:Four years after its release, Kubernetes has come a long way by pacala on Wednesday June 06, 2018 @05:40PM (#17251328)
Name one.
Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @07:22PM (#17251985)
You mean a failover solution? It's hard to give a list here because it is such a large space and it completely depends on your product/application. It is more like a category of use cases than one specific use case.

Some ideas for things you could do in a web/website/internet context assuming you have a single point of presence:

One type of "HA" is network-level failover; haproxy (L7), nginx (L7) and pacemaker (usually L3) seem to be very popular options, but I think there are dozens of other alternatives. In terms of network-level failover, things get more interesting once you are running in multiple locations, have more than one physical uplink to the internet or do the routing for your IP block yourself.

For application-level failover and HA, one option for client/server style applications is to move all state into a database which has some form of HA/failover built in (so pretty much every DB). I think this is very common for web applications and also for some intranet applications.

Assuming you have a more complex application running in a datacenter, there is also a lot of very interesting stuff you can do using a lock service like Zookeeper or etcd or really almost any database. Of course, you can also make your app highly available without using an external service; there is a mountain of failover/HA research and algorithms that you can put directly into your app (2pc, paxos, raft, etc). Of course all of these require some cooperation/knowledge from the application. For some apps it might be very hard to make them "HA" without relying on an external system, but for some apps it will be trivial.

Note that when you move away from a web/datacenter context, to something like telecommunication or industrial automation, so something that doesn't run as a process in a datacenter but is implemented {as, on} hardware in the field, failover and high availability will have an entirely different meaning and will be done in a totally different way.

Re:Four years after its release, Kubernetes has come a long way by peterwwillis on Wednesday June 06, 2018 @06:45PM (#17251747)
Most single-instance, single-zone failover scenarios can be handled with shell scripts, the AWS API, and cron.

But the parent's comment is missing the point. K8s is not for failover. K8s is literally just a giant monolith of microservices for running microservices. It's not intended to provide failover for non-microservices, it's intended only to run microservices, and as a side-effect of needing to be able to scale them, it inherently provides some failover mechanisms.

Re:Four years after its release, Kubernetes has come a long way by pacala on Wednesday June 06, 2018 @07:09PM (#17251912)
Given GKE or AKS, not quite sure how "shell scripts, the AWS API, and cron" is "much simpler" than:

   $ docker build . -tag gcr.io/google-samples/hello-app:1.0
   $ docker push gcr.io/google-samples/hello-app:1.0
   $ kubectl run hello-server --image gcr.io/google-samples/hello-app:1.0 --port 8080
   $ kubectl expose deployment hello-server --type "LoadBalancer"
where the Dockerfile content is:

   FROM golang:1.8-alpine
   ADD . /go/src/hello-app
   RUN go install hello-app

   FROM alpine:latest
   COPY --from=0 /go/bin/hello-app .
   ENV PORT 8080
   CMD ["./hello-app"]
https://cloud.google.com/kubernetes-engine/docs/quickstart

https://github.com/GoogleCloudPlatform/kubernetes-engine-sam...

Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @07:45PM (#17252102)
The difference in simplicity is not in the interface that is presented to you as a user. The difference is that your shell script will have a couple hundred lines of code, while the docker and kubectl commands from above will pull in literally hundreds of thousands of lines of additional code (and therefore complexity) into your system.

I'm not saying that is a bad thing by itself, but there definitely is huge amount of added complexity behind the scenes.

Re:Four years after its release, Kubernetes has come a long way by tannhaeuser on Wednesday June 06, 2018 @08:17PM (#17252258)
That's nice and all as long as it works. If there are any problems with it (network, auth, whatever) have fun with even diagnosing the bottomless complexity behind your innocous lil' kubectl command.
Re:Four years after its release, Kubernetes has come a long way by pacala on Wednesday June 06, 2018 @08:40PM (#17252348)
The error messages are quite good. There are extensive logs. There are plenty of probes in the system. Just the other day we `kubectl exec <pod_name> /bin/bash` to test network connectivity for a new service.

To the best of my exposure, Kubernetes is a well engineered system.

Re:Four years after its release, Kubernetes has come a long way by tannhaeuser on Wednesday June 06, 2018 @09:02PM (#17252439)
The point is that automation software for admin tasks is a zero-sum game: the more a tool does, the less your devops staff will be able to leverage their know-how. The more magic your orchestration does, the less your ops will be able to fix prod problems. And for getting anything done with k8s you'll need a whole staff of very expensive experts.
Re:Four years after its release, Kubernetes has come a long way by peterwwillis on Wednesday June 06, 2018 @11:24PM (#17253012)
Look at it this way. Kubernetes is a Tesla Model X, and scripts/aws/cron is an electric scooter.

You can try to go cross-country with either of them. One was engineered to survive extreme environments, protect you from the elements, and move really fast. The other was engineered to leave you exposed, operate in a moderate climate, and go much slower.

If you have problems with the former, it's time to call the technicians. If you have problems with the latter, you might fix it yourself with a pocket screwdriver.

Re:Four years after its release, Kubernetes has come a long way by aphexairlines on Thursday June 07, 2018 @03:43AM (#17254019)
CodeDeploy has been doing this for ages.

https://docs.aws.amazon.com/codedeploy/latest/userguide/gett...

Distelli (now Puppet Pipelines) too:

https://www.youtube.com/watch?v=ZZlYADohS4Q

And then there are the PaaS options like Heroku, Beanstalk, AppEngine, Lambda, Serverless, and Apex.

Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @07:43PM (#17252088)
> But the parent's comment is missing the point

That was my point. I wanted to point out that while some people have only/first heard about failover in the context of kubernetes, it is not something that is specific to kubernetes or even the problem that kubernetes was build to solve.

Of course it is not designed to be a failover solution specifically and using it (exclusively) as such would be ill-advised; I was just trying to be diplomatic while pointing that out.

Re:Four years after its release, Kubernetes has come a long way by jontro on Wednesday June 06, 2018 @05:59PM (#17251454)
We're using elastic beanstalk currently (multi docker conatiner). Not that I'm advocating it and I'm really interested in k8s now that aws has eks, but ebs is really simple to use for a simple setup.
Re:Four years after its release, Kubernetes has come a long way by tlarkworthy on Wednesday June 06, 2018 @04:51PM (#17251008)
I don't think that's true. If you are using bash or chef to interact with the cloud then k8s can replace those with a much cleaner model.

My biggest fear is helm becomes chef cookbooks

Re:Four years after its release, Kubernetes has come a long way by ridruejo on Wednesday June 06, 2018 @05:07PM (#17251133)
Can you elaborate on the concerns with helm?
Re:Four years after its release, Kubernetes has come a long way by temikus on Wednesday June 06, 2018 @06:07PM (#17251498)
It’s probably referring to people using Chef cookbooks blindly without any knowledge of how to configure or maintain the service itself.

I have interacted with people who run production workloads via Helm but struggle to do very basic things with kubectl (like recreating a service).

This is an argument that can be used for any high level abstraction, however. I guess Chef is used since they have very good testing tools so Cookbooks are relatively more stable than other CM definitions. Until things go wrong with the service that is.

Re:Four years after its release, Kubernetes has come a long way by tlarkworthy on Wednesday June 06, 2018 @06:18PM (#17251590)
That community sourced cookbooks are security nightmares or under maintained. I cannot see how helm charts can stay quality without a curated app store.
Re:Four years after its release, Kubernetes has come a long way by ridruejo on Wednesday June 06, 2018 @07:16PM (#17251947)
Thanks for the clarification. The official Helm charts go through a thorough community review process. We (Bitnami) are committed to making high-quality, up-to-date Helm charts. You can check https://hub.kubeapps.com
Re:Four years after its release, Kubernetes has come a long way by tlarkworthy on Wednesday June 06, 2018 @10:56PM (#17252901)
I would be so happy if you pulled it off
Re:Four years after its release, Kubernetes has come a long way by geggam on Wednesday June 06, 2018 @06:28PM (#17251646)
Any unmaintained infrastructure as code piece will quit working.

Success I have had with chef has been generating my own cookbooks that dont try to do everything for every OS

Re:Four years after its release, Kubernetes has come a long way by tlarkworthy on Wednesday June 06, 2018 @10:57PM (#17252905)
Yes exactly. Everything tends towards rolling your own.
Re:Four years after its release, Kubernetes has come a long way by outside1234 on Wednesday June 06, 2018 @04:37PM (#17250882)
You probably don't need the cloud either, but everyone recognizes that it is a productivity gain. To me, this is the analogous case for K8s, you can probably do the same with VMs, but everything is a lot more productive with containers and K8s.
Re:Four years after its release, Kubernetes has come a long way by jeffreyrogers on Wednesday June 06, 2018 @04:54PM (#17251026)
I worked at a place that was deploying K8s as I was leaving and my experience was that K8s was complicated and no one knew exactly how to use it or why we should use it, other than it seemed to be what everyone else was doing. I didn't notice any operational improvements over our prior system (which was largely ad hoc), but the kinks may have been worked out by now.
Re:Four years after its release, Kubernetes has come a long way by pm90 on Wednesday June 06, 2018 @05:01PM (#17251092)
If you read the Innovators Dillemma, there is a very clear chapter about how new technologies don't really deliver their promises immediately; but as they're adopted and operators/users learn how to work with it, that's when productivity changes are very much visible. It took quite some time for AWS to be mainstream and then to be adopted and operated in a way that delivered high productivity gains, and it seems to be the same for k8s right now.
Re:Four years after its release, Kubernetes has come a long way by zaphirplane on Wednesday June 06, 2018 @07:29PM (#17252023)
I really wonder about the causality of that, is it changing things to adapt the new technology makes you clean up / refresh / fix known issues along the way or is it the new technology itself causes less issues down the track
Re:Four years after its release, Kubernetes has come a long way by pm90 on Wednesday June 06, 2018 @08:03PM (#17252192)
That doesn't really make sense. If you're moving to a new technology, even if you clean up/refresh/ fix known issues along the way, you are likely to accumulate similar such issues in the new technology as well, which would lead to a decrease in productivity. But that has never really been observed.

While I agree that the cleanup can inevitably lead to productivity gains, I believe they are overshadowed by the productivity gains from the actual change that is being made.

Re:Four years after its release, Kubernetes has come a long way by tannhaeuser on Wednesday June 06, 2018 @08:41PM (#17252352)
That only makes sense when seen as an obstacle to initial adoption from an innovators and sales point of view and could be said just about every innovation. But obviously not every IT innovation makes it.
Re:Four years after its release, Kubernetes has come a long way by asdfman123 on Wednesday June 06, 2018 @05:19PM (#17251213)
It depends on how frequently you need to spin up new containers.

As someone on Azure, I think the solution for small-ish guys like me (who still need like 8 VMs in production to run our site due to special requirements) is a greatly simplified container management system. Something that fulfills the basic Kubernetes features, but isn't such a steep learning curve.

I think Microsoft is doing this with Azure Container Instances, but I haven't tried it yet. (No doubt AWS is doing something about it as well.)

Or they're going to take care of all the k8 configuration crap through some neat Visual Studio Tools. In a VS preview I saw a "deploy to Kubernetes" button. I just want something that will give me more web API instances when my server is getting slammed.

Re:Four years after its release, Kubernetes has come a long way by mountainofdeath on Wednesday June 06, 2018 @05:43PM (#17251349)
AWS ElasticBeanstalk or Google App Engine flex or something from Heroku is what you are talking about. Container management but simplified for the common use case. You can implement the same thing with k8s, but why bother for small stuff. The real value in k8s is large, company-wide installations where everyone shares each others hardware. Even then, I've heard anecdotes that kubernetes doesn't scale to the point Borg or whatever other proprietary system does. Then again, those are niche use cases to begin with.
Re:Four years after its release, Kubernetes has come a long way by pmh on Wednesday June 06, 2018 @05:43PM (#17251351)
Unless they've reduced the price, Azure Container Instances is mainly for one-off/bursty tasks. If you need a service running 24/7, it's prohibitively expensive.

I think once their managed kubernetes (AKS) hits GA, the learning curve will be slightly lower.

Re:Four years after its release, Kubernetes has come a long way by slow_donkey on Wednesday June 06, 2018 @07:05PM (#17251884)
Right, container instances isn't equivalent to deploying a vm with coreos like gcp offers.

We've been running off aks for a bit, the only negative is how slow spinning up new clusters can take.

Re:Four years after its release, Kubernetes has come a long way by dorfire on Sunday June 10, 2018 @03:01AM (#17276965)
Re:Four years after its release, Kubernetes has come a long way by bartread on Wednesday June 06, 2018 @04:53PM (#17251022)
> You probably don't need the cloud either, but everyone recognizes that it is a productivity gain.

Care to quantify that? Because we don't use the cloud (in fact, in my current role, we don't even make use of virtualisation for production infrastructure) and we have no issues with productivity. Or rather, let's be a little more honest, we have no issues with productivity that would be solved by a move to AWS/Azure/$_some_other_cloud_provider.

Re:Four years after its release, Kubernetes has come a long way by mafro on Thursday June 07, 2018 @01:18AM (#17253460)
Just because you "have no issues with productivity", doesn't necessarily mean that you couldn't be more productive. In my experience it takes a move to the cloud to actually see where those gains lie.

Disclaimer: I have no idea about your actual real-life situation. This is vague and hand wavy on purpose.

Re:Four years after its release, Kubernetes has come a long way by bartread on Thursday June 07, 2018 @02:16AM (#17253702)
> In my experience it takes a move to the cloud to actually see where those gains lie.

Whilst I realise you were being vague on purpose, the above statement represents no kind of business justification.

Re:Four years after its release, Kubernetes has come a long way by mafro on Thursday June 07, 2018 @04:48PM (#17260117)
Indeed it is not meant to be any kind of business justification - I clearly cannot comment your business and its needs :)

In my experience of moving applications to the cloud, there are always folks who don't believe there is any gain to be made. These folks have always been wrong (so far!)

I suppose I'm only commenting because I hate to see someone snub the potential of cloud. But hey I'm just some guy on HN.

Re:Four years after its release, Kubernetes has come a long way by bartread on Thursday June 07, 2018 @07:21PM (#17261258)
The thing is, in previous positions we've used the cloud plenty, and it's been fine. But for what I'm doing now? No way. Just not the right fit. I therefore become extremely sceptical when people cite $unspecified_but_awesome_gains from switching.
Re:Four years after its release, Kubernetes has come a long way by mafro on Thursday June 07, 2018 @07:28PM (#17261292)
That's awesome. Can you elaborate more on your setup, and why cloud doesn't make sense? I'm really interested in cases like this.
Re:Four years after its release, Kubernetes has come a long way by outside1234 on Wednesday June 06, 2018 @06:27PM (#17251634)
You forgot about all the folks managing / maintaining / cabling / air conditioning all your servers. That is the productivity gain you are missing.
Re:Four years after its release, Kubernetes has come a long way by jjeaff on Wednesday June 06, 2018 @06:52PM (#17251799)
I doubt they are running their own data center. Probably just renting dedicated boxes.
Re:Four years after its release, Kubernetes has come a long way by bartread on Thursday June 07, 2018 @02:14AM (#17253694)
> Probably just renting dedicated boxes.

Absolutely correct.

Re:Four years after its release, Kubernetes has come a long way by cc81 on Wednesday June 06, 2018 @04:52PM (#17251015)
I don't have much practical experience with Kubernetes but I think one argument is that there are other alternatives to it that might give equal or better productivity gains for a lot of people.

For example Azure App Service or equivalent from the other vendors. You don't have to manage a VM.

Re:Four years after its release, Kubernetes has come a long way by slow_donkey on Wednesday June 06, 2018 @07:10PM (#17251920)
App service is alright but I'd recommend Heroku over app service any day. However, there is some visible work being done to improve app service like supporting docker compose files to run multiple containers but the basics are still very painful.

Most notably, swapping servers (slots) always caused downtime, accessing logs was flaky and slow, azure interface is just plain slow, continuous deploy was flaky (or it just didn't kick in until 20 min later?) and I had to manually restart servers many times

Re:Four years after its release, Kubernetes has come a long way by swyx on Wednesday June 06, 2018 @04:45PM (#17250955)
i am a mainly js engineer and so far have lived entirely within the firebase world as far as my backend goes. in your (and everyone else's) opinion, then: when -do- you need it? set some quantifiable bars for me please?
Re:Four years after its release, Kubernetes has come a long way by tnolet on Wednesday June 06, 2018 @04:53PM (#17251020)
I could throw the "you'll know it when you see it" but I'll try not to! 1. You have a large amount of underused hardware/VM 's you want to cluster. 2. You have an eloborate microservices system with a dynamic service discovery mechanism. 3. You have the above over many regions / teams. 4. All your services are mostly stateless. 5. Your team of rockstar ops engineers are building their own crappy version of Kubernetes just to deal with the engineering challenges your business faces.
Re:Four years after its release, Kubernetes has come a long way by cogman10 on Wednesday June 06, 2018 @07:02PM (#17251866)
Funnily, my current employer is just that. We predate K8s, so it can be forgiven somewhat. Transitioning, though, is just awful. We have every reason to transition, yet getting the system setup, running, and working with the current infrastructure is a daunting task.

You don't need K8s, until you do, at which point switching can be a nightmare scenario.

Re:Four years after its release, Kubernetes has come a long way by swyx on Thursday June 07, 2018 @12:16AM (#17253243)
thanks, i'm pretty far from that world then, haha. appreciate the list of concerns, i hope to someday worry about these things.
Re:Four years after its release, Kubernetes has come a long way by numbsafari on Wednesday June 06, 2018 @05:43PM (#17251354)
If you are able to be cost effective and run services on firebase, then keep doing so. Please!

If you need to do something that doesn’t map well to firebase, then my next stop would be AppEngine.

If what you want to run won’t map well to AppEngine, then start looking at K8S.

For me, this is usually wanting to run some kind of legacy system, or where a higher level of security is needed, or if you want or need to plan to run across multiple cloud vendors or in a mixed workload environment.

If AppEngine has been covered by google’s BAA when I started my current gig, I would have just used that, with k8s to handle the couple of oddball things that we are using that wouldn’t run as well there.

Personally, it’s not about scale so much as certain kinds of architectures or problems. You might have a system that needs to run a smallish number of nodes, but has reasons for needing what kubernetes offers without wanting to reinvent the wheel yet again with the existing cast of characters.

Re:Four years after its release, Kubernetes has come a long way by HandMadeTail on Wednesday June 06, 2018 @08:59PM (#17252432)
But what do you do if your product needs to potentially scale to include China?

Firebase, AppEngine, etc. will all be blocked there?

Re:Four years after its release, Kubernetes has come a long way by numbsafari on Thursday June 07, 2018 @06:17AM (#17254620)
Honestly, I have no idea about Firebase or AppEngine in China. It might entirely be FUD. But, if what you imply is true, then Kubernetes is probably a good solution for your goals.

Run your application on GKE and grow your company. When it comes time to move into China, add a cluster on a cloud provider there.

Re:Four years after its release, Kubernetes has come a long way by swyx on Thursday June 07, 2018 @12:17AM (#17253245)
love the progressive recommendations here. looks like my next stop is AppEngine!
Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @05:13PM (#17251176)
When you notice that you spend such significant money (i.e. multiple salaries) on hardware capacity planning that employing a team to operate kubernetes might be cheaper instead. Probably not going to happen unless you run on hundreds to thousands of machines.
Re:Four years after its release, Kubernetes has come a long way by barrkel on Wednesday June 06, 2018 @06:59PM (#17251844)
Or vmware licensing is killing you and you can't use PaaS for contractual reasons.
Re:Four years after its release, Kubernetes has come a long way by kross on Wednesday June 06, 2018 @09:49PM (#17252614)
Polyglot (multiple tech/languages), unknown scale, dramatic flash traffic or potential thereof.

Really your next step in general is understanding containers and where they might be useful. K8s consideration is probably after that.

Re:Four years after its release, Kubernetes has come a long way by avip on Wednesday June 06, 2018 @07:03PM (#17251874)
What's your bar for using webpack (or your js builder of choice)? If you're fluent and at-ease with it, you'll probably use it even for hello-world app. Same (in spirit) for k8.
Re:Four years after its release, Kubernetes has come a long way by stemcc on Wednesday June 06, 2018 @04:36PM (#17250875)
In your experience with these deployments, which K8s features aren't needed? Is it the orchestration as such that's not necessary, or K8s approach wrt to the workloads deployed?
Re:Four years after its release, Kubernetes has come a long way by tnolet on Wednesday June 06, 2018 @04:42PM (#17250927)
Sorry for not being clear. There is nothing wrong with K8s features. It solves a specific problem really well. However, most people & companies I interact with don't have that problem. Or that problem is just a very marginal piece of their full set of problems. Running K8s, maintaining it and making your apps work with it is pretty big undertaking. Only go there if the upside is so great it blows any alternative away.

I guess the popular term is "cargo culting". Or maybe Heroku is not hip anymore although it probably is the best advice for many companies.

Re:Four years after its release, Kubernetes has come a long way by meddlepal on Wednesday June 06, 2018 @04:54PM (#17251033)
I strongly disagree. I've built a couple in house PaaS type systems in the past with various degrees of success on more classical tech like straight up EC2 using a variety of Terraform/Ansible/CloudFormation/Scripting but the dev experience has always been painful compared to Kubernetes.

Developers really love working with Kubernetes. It doesn't take a lot to get started and if you're not doing anything too crazy it is a huge productivity boon IMO.

Where things get hairy is persistence, but that's the case regardless of Kubernetes or not.

Re:Four years after its release, Kubernetes has come a long way by auslander on Thursday June 07, 2018 @12:21AM (#17253265)
>Terraform/Ansible/CloudFormation/Script

This number of tools speaks for itself. Yeah, you would be better off with out of the box stuff due to lack of skills.

Re:Four years after its release, Kubernetes has come a long way by meddlepal on Thursday June 07, 2018 @07:39AM (#17254914)
Ah yes, you're right. I am sure you have a much better understanding of the requirements each of these build outs sitting behind your keyboard over there.
Re:Four years after its release, Kubernetes has come a long way by rvanmil on Wednesday June 06, 2018 @04:55PM (#17251039)
Heroku is still amazing indeed. Our team recently moved all Node.js based systems from AWS to Heroku and it has given us a tremendous productivity gain and reduction in complexity. Before we made that decision we spent about 8 months experimenting with K8s, only to discover it actually made things way more complex for us and reducing productivity without any clear benefits.
Re:Four years after its release, Kubernetes has come a long way by ShabbosGoy on Wednesday June 06, 2018 @06:00PM (#17251456)
In that specific case, Kubernetes is approaching YAGNI territory. But as others have stated, it’s easy to get started with something like Amazon EKS and not have to worry about a thing.

Edit: Put another way, give me APIs instead of config file hell.

Re:Four years after its release, Kubernetes has come a long way by rb808 on Wednesday June 06, 2018 @04:43PM (#17250936)
> I know of at least one hot startup deploying Kubernetes just to attract engineering talent.

Interesting proposition - if it helps attract better talent does that mean they needed k8s?

Re:Four years after its release, Kubernetes has come a long way by collyw on Thursday June 07, 2018 @04:26AM (#17254218)
Sadly I doubt it.

Look at this question and top answer from Quora. It asks how you would build a website.The answer dives into a host of engineered technologies that would suit something you expect to need to scale massively.

https://www.quora.com/If-you-were-to-build-a-website-right-n...

Re:Four years after its release, Kubernetes has come a long way by tnolet on Thursday June 07, 2018 @10:57AM (#17256284)
Wow. The top answer on that one is terrible. I would advise the exact opposite. Istio!? Nice idea, but have you tried using it? That is some serious alpha stage software almost noone needs till very late in their tech maturity.
Re:Four years after its release, Kubernetes has come a long way by brodsky on Wednesday June 06, 2018 @04:37PM (#17250886)
what would you recommend to use for container orchestration and service discovery instead of Kubernetes?
Re:Four years after its release, Kubernetes has come a long way by oldmanhorton on Wednesday June 06, 2018 @04:39PM (#17250905)
I believe the parent comment was trying to say that most people don't need service orchestration or discovery
Re:Four years after its release, Kubernetes has come a long way by tnolet on Wednesday June 06, 2018 @04:44PM (#17250940)
Yes. That is exactly what I meant. Mea culpa for not being more clear.
Re:Four years after its release, Kubernetes has come a long way by pacala on Wednesday June 06, 2018 @04:57PM (#17251056)
My team runs a grand total of 1 server, 1 daemon and up to 5 concurrent batch jobs. Sometimes we bring up 1 extra server for experimentation.

It is handy to not break the bank and use GKE autoscaling for the batch jobs. Surely we could setup some batch job manager over some raw VMs, or maybe spend some time to select and learn to use yet-another AWS service.

It is handy to go to the GKE "Workloads" page and see how many servers are currently active, using how many pods. Surely we could pull that off with raw VMs, but it would take me a few minutes extra / a brittle script to pull it off.

Do we _need_ Kubernetes? No. It makes us more productive? Yes. Am I confident that in the next assignment I'll be able to reuse my skills, regardless of what VM provider they use? 90% yes, pending EKS becoming usable.

Echoes of "you don't need C, just use Assembly, it's simpler and faster" from early '90s gaming industry.

Re:Four years after its release, Kubernetes has come a long way by brodsky on Wednesday June 06, 2018 @04:51PM (#17251004)
no worries; that makes sense and I agree with you.

(it's strange that my question resulted in downvotes).

Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @04:55PM (#17251037)
I think the recommendation is not to use another orchestration service, but to keep the infrastructure simple and not use any orchestration or service discovery at all.

Adding service discovery and container orchestration will probably not make your product better. Instead it will add more moving parts that can fail to your system and make operations more complex. So IMO a "containerized microservice architecture" is not a "feature" that you should add to your stack just because. It is a feature you should add to your stack once the benefits outweigh the costs, which IMO only happens at huge scale.

Most people know that "Google {does, invented} containers". What not so many developers seem to realize is that a Google Borg "container" is a fundamentally different thing from a Docker container. Borg containers at Google are not really a dependency management mechanism; they solve the problem of scheduling heterogenous workloads developed by tens of thousands of engineers on the same shared compute infrastructure. This however, is a problem that most companies simply do not have, as they are just not running at the required scale. At reasonable scale, buying a bunch of servers will always be cheaper than employing a Kubernetes team.

And if you do decide to add a clever cluster scheduler to your system it will probably not improve reliability, but will actually do the opposite. Even something like borg is not a panacea; you will occasionally have weird interactions from two incompatible workloads that happen to get scheduled on the same hardware, i.e. operational problems you would not have had without the "clever" scheduler. So again, unless you need it, you shouldn't use it.

I think the problem that Docker does solve for small startups is that it gives you a a repeatable and portable environment. It makes it easy to create an image of your software that you are sure will run on the target server without having talk to the sysadmins or ops departments. But you don't need kubernetes to get that. And while I can appreciate this benefit of using docker, I still think shipping a whole linux environment for every application is not the right long-term way to do it. It does not really "solve" the problem of reliable deployments on linux; it is just a (pretty nice) workaround.

Re:Four years after its release, Kubernetes has come a long way by barrkel on Wednesday June 06, 2018 @07:04PM (#17251878)
Without service discovery, you end up generating configuration all over the place just to wire together your services, letting everybody know about which IP address and port where - super tedious. Add in multi-tenancy, and you're in a bunch of pain. Now try and take that setup and deploy it on-prem for customers that can't use cloud services - you rapidly want something to abstract away the cluster and its wiring at the address allocation level.
Re:Four years after its release, Kubernetes has come a long way by firebacon on Wednesday June 06, 2018 @07:58PM (#17252166)
I'm not sure I understand this. You have a product that is split into many different components, and when you deploy this product to a customer site, each component runs on different hosts, so you have a bunch of wiring up of service addresses to do for every deployment?

Could something like mDNS be a lightweight solution to that problem?

And also I am genuinely curious how kubernetes would solve that. When you install kubernetes on all of these machines, don't you have to manually do the configuration for that either? So isn't it just choosing to rather configure kubernetes instead of your own application for each deployment? If it is that much simpler to setup a kubernetes cluster than your app, maybe the solution is to put some effort into the configuration management part of your product?

Re:Four years after its release, Kubernetes has come a long way by kross on Wednesday June 06, 2018 @10:11PM (#17252725)
I'm not sure you understand the value k8s proposes based on your comments throughout this entire thread.

Managing many nodes is the reason orchestration software exists. Your suggestion to "put some effort into configuration management" is effectively naive, homegrown orchestration.

Build or buy? That's the same argument - except k8s is free and available from many providers.

Re:Four years after its release, Kubernetes has come a long way by shaklee3 on Wednesday June 06, 2018 @10:17PM (#17252742)
Kubernetes service Discovery is just dns, so it sounds like you're doing the same thing, but by exiting CNAMEs.
Re:Four years after its release, Kubernetes has come a long way by clhodapp on Friday June 08, 2018 @01:28AM (#17262825)
Ah, if you look under the covers, Kubernetes service discovery is actually kernel-space virtual load balancers running on each node in your cluster. The DNS record just points to the "load balancer".
Re:Four years after its release, Kubernetes has come a long way by dbmikus on Thursday June 07, 2018 @12:25AM (#17253281)
If your docker containers all build off of the same OS and then you run them on a host using that OS, won't the containers share the lower filesystem layers with the host?
Re:Four years after its release, Kubernetes has come a long way by aphexairlines on Thursday June 07, 2018 @04:06AM (#17254116)
Amazon describes its internal service architecture here:

https://www.youtube.com/watch?v=dxk8b9rSKOo

Services use an internal tool called Apollo to describe the service, deploy it to EC2 VMs, and auto-scale. Apollo inspired AWS CodeDeploy and AWS AutoScaling.

Services are reachable via load balancer sets similar to AWS ELB/ALB/NLB.

You reach a service by the LB-set's DNS name.

If you don't want to use AWS or AWS-specific deployment tools, you could use Puppet Pipelines to do the same on Google Cloud or Azure. Puppet Pipelines was built by people who previously built some of those internal Amazon tools and offers similar functionality but cross-cloud.

And if you want even fewer moving parts, just go PaaS or serverless.

Re:Four years after its release, Kubernetes has come a long way by auslander on Thursday June 07, 2018 @08:36AM (#17255203)
Never heard of Apollo, had to dig :) In short, it is a predecessor of AWS CodeDeploy.

www.allthingsdistributed.com/2014/11/apollo-amazon-deployment-engine.html

Re:Four years after its release, Kubernetes has come a long way by AndyNemmity on Wednesday June 06, 2018 @06:42PM (#17251732)
Depends on your definition of "need", it enables me to manage a ton of stuff across multiple dcs and regions easily. It would feel like the dark ages to be without it as this point.