This is such nonsense not to mention hilariously hypocritical since you criticise "chasing the latest hotness" but then advocate for a Serverless architecture.

Anyway go and try and build a typical application with Lambda/SAM. It is a nightmare of complexity and all you are doing is moving your logic to AWS where you pay 100x the cost of just running it yourself in a container.

And the idea that Kubernetes isn't mature is pretty laughable. It's used everyday by Netflix, eBay, Apple, Microsoft, IBM, Lyft, Uber, Square, Google, Pinterest, Stripe, Airbnb, Yahoo, Salesforce etc. And with AWS you have EKS which allows you to run containers in a HA and managed way.

If you are not married to AWS, try GKE.

EKS is still... early. I'd rather manage myself with Kops or Terraform in its current stage. And in fact, that's what we do at my company.

We tried using EKS. Unfortunately, HorizontalPodAutoscalers are configured to use metrics-server and it is not currently possible to run metrics-server on EKS. Had to replace it with a kops + terraform setup.

No project with 759 confirmed bugs on Github should be considered mature, in my opinion.

This is a nonsensical metric and you know it. How many of these are production impacting? For me, that just reflects a lot of activity. Take a look at how many get resolved.

I manage a few hundred worker nodes, they are heavily loaded at all times but happily chugging along. My pager is silent and has been for a while now. Last k8s issue we saw was actually our own fault, due to a misconfiguration.

I guess it is a little silly. But it remains a complicated project under heavy development.

I'm the kind of guy who enjoys Python 2.7 because it's deprecated, it's done changing :)

I try remember the maxim, Happiness is a Boring stack.

Ha. EKS is a joke. I have written services powered by lambda. As a matter of fact I’ve used AWS/GCP/Azure. I mentioned lambda as an example of a fully managed service - it was not to “endorse” serverless.

Also if you look at a container long enough and squint you will see the words serverless emerge.

I also had the “pleasure” of working with K8S. If anything, this looks like a pretty good play from Google to get you to eventually run your workloads in GCP.

> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Yes there is lock-in. Yes, if not properly arhitected it can be a mess.

I've just rolled off a project (line of business web app + cluster of workers for background job processing), this more or less describes what we have running.

Some of the architecture is a bit wrong (it wasn't designed for AWS but shifted there after it had been running for a year or so) but the system works well enough to deliver value to the business.

As a developer I hate state, things that aren't properly isolated, ill-defined system boundaries -- but it's not obvious to me what the business case would be to containerise everything.

> it's not obvious to me what the business case would be to containerise everything

Containers allow you to move apps trivially between environments and guarantee that they will just work. It allows you to isolate dependencies between apps e.g. Python 2 versus Python 3. It allows you to move apps between cloud providers or between on premise and cloud. With platforms like Kubernetes it allows you to easily scale and self heal when nodes die.

And compared to rewriting your app in Lambda which is expensive and complex it is simple to build a container as almost every language has automated tooling.

> Containers allow you to move apps trivially between environments and guarantee that they will just work.

Assuming your staff can keep your cluster un-screwed, am I right?

Technologies like vmware, (which also give you os independence not just linux flavors) if we're talking on premises also allow you to move apps between environments. It's trivial to, gasp, push a machine out of vmware player to the cloud even. Pick your poison, vmware, virtualbox, azure, aws, gcp.

I'll run kubernetes as provided by GCP or AWS for my clients if it's warranted (ohhh, you wanted that in "webscale", got it. ;) ), but I really feel sorry for all of the on-prem "enterprise" shops that have taken the hype bait and are now paying the maintenance burden under all of the false pretenses that are flying around. "Horror's of Upgrading Etcd Beneath Kubernetes" with dozens of production application instances, with uptime SLA's and real customer business impact? Indeed. Fraught with peril, and without the right staff onboard, disastrous.

Folks be like, hey a team of consultants just finished building out our new kubernetes cluster, now we want to run our mission critical oracle/mssql servers on it. They said it should work "fine". Too bad they all got jobs at insert mega capacity company here right after we cut the invoice.

Y'all remember when everyone would line up to get the latest microsoft windows beta? Flashback. Yo yo yo, XML is all the rage! Not. Relational databases are dead! Um, no.

Maybe this needs some time to travel down and back up the hype cycle curve. Power to all of you beta testers! You're truly doing gods work :)

> Folks be like, hey a team of consultants just finished building out our new kubernetes cluster, now we want to run our mission critical oracle/mssql servers on it. They said it should work "fine". Too bad they all got jobs at insert mega capacity company here right after we cut the invoice.

mmm!

http://classicprogrammerpaintings.com/post/131118350394/army...

http://classicprogrammerpaintings.com/post/169578560497/rhel...

http://classicprogrammerpaintings.com/post/153774064947/frid...

http://classicprogrammerpaintings.com/post/142748219994/serv...

Thanks shoo, that made me chuckle - maybe I'm feeling a bit too angsty this Friday evening.

I love all of the new development, we're going great places, we just need to prescribe the right medicine for the patients so to speak. Cocaine was great in coca-cola before folks realized what it was doing to people. :)

I really love how Dan McKinley frames this kind of thing in terms of each company having a budget of "innovation tokens".

http://mcfunley.com/choose-boring-technology

edit: now it's a club! http://boringtechnology.club/

> Let's say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while.

> If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens. If you choose to use service discovery tech that's existed for a year or less, you just spent one of your innovation tokens. If you choose to write your own database, oh god, you're in trouble.

> Any of those choices might be sensible if you're a javascript consultancy, or a database company. But you're probably not. You're probably working for a company that is at least ostensibly rethinking global commerce or reinventing payments on the web or pursuing some other suitably epic mission. In that context, devoting any of your limited attention to innovating ssh is an excellent way to fail.

> There is technology out there that is both boring and bad. You should not use any of that. But there are many choices of technology that are boring and good, or at least good enough. MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.

This +1000. :) People, remembering to focus on core-competencies, making the world a better place.

This is why I'm growing into a bit of a fan of the hashicorp suite at the moment. It allows for a gradual, problem-driven extension of an infrastructure, instead of requiring you to start over. At our place we did a couple of iterations:

- First off we went from manually managed servers to chef managing servers. That was good progress, because it allowed us to scale a growing application on a cloud provider due to a large new contract.

- Then we added vault in order to simplify secret generation, management and rotation in chef. It's cool, because now we have a secure secret storage. We can give our devs access specific access to the secrets of clusters they manage but not other clusters. We can script a lot of stuff around vault.

- Then we added terraform to manage VMs easier. We should have done that earlier, I suppose, but hindsight.

- And now our devs are having large issues with their docker-based test setups, so we can open up the consul cluster and deploy nomad for this use case. We'll probably migrate some other services into that nomad cluster so we can get them loadbalanced with little effort. We'll probably shuffle some annoying things in chef around and use consul-template there.

I like that approach, because it is problem-driven and converges to simplify existing problems. For example, we have an elastic stack, and we won't move the elasticsearch cluster or the influxdbs around it away from chef on bare metal. It's a solid and stable setup, why change it.

Have an upvote. Appreciate the pragmatism :)

Not sure I agree. If you need a container to do the things you mention you’re already in a pretty bad shape.

Not to mention that a lot of people don’t understand what a container is.

> Not to mention that a lot of people don’t understand what a container is.

That's part of the appeal, the abstraction away of all those pesky, irrelevant details. Developers just want their app to run, as has been mentioned elsewhere in the thread. That desire is understandable.

So long as the abstraction isn't too leaky and nothing underneath breaks, there's no downside. It's all upside in terms of human productivity and time to market.

Even introduced inefficiency (if any) is unimportant if VC money is fueling the auto-scaling. There's a popular aphorism about premature optimization. Humans are also, in general, far more expensive than machines, especially at scale, and even a mature/traditional company would ignore this at their peril.

If problems do eventually crop up with containers or the toolsets around them, chances are, by then, their sheer popularity will ensure the availability of a cadre of experts who can troubleshoot. They may even understand what a container is, even if it's terribly unfashionable to admit, as is the case with operating systems today.

Every new technology/tool has some growing pains and is subject to what some consider misuse (due to ignorance). That doesn't necessarily mean it's best to reject it outright.

Big fan of playing with new tech. In a safe environment where failure is an option. Don’t go betting the farm on shiny things.

"Personally" (professionally), neither am I. This kind of conservatism was formed after decades of sysadmin/ops experience.

However, having acquired a strong affinity for startups, I also accept that risk (even complete, betting-the-farm risk) is totally OK, so long as that risk is taken in an informed manner. Startups, especially early ones, are a risky proposition from the get-go, and VC money tends to amplify that.

The market doesn't much punish a web startup that grew like crazy but lost some user-generated content by using some "NoSQL" database when it first came out. It does, however, punish the one that failed to grow by being too conservative.

That's obviously a false dichotomy, but I believe that's essentially the perception that's created, partly by characterizing new technologies as dangerous because they're new and "shiny".

Ultimately, I consider mine to be a service profession and an engineering one. As such, if I think a new tech is too undesirable, it's up to me to provide an alternative that actually addresses the original problems (without offloading the burden onto my users/customers).

> Not to mention that a lot of people don’t understand what a container is.

Well, I'll bite. Do you? What is a container?

Best definition: there is no such thing as a container. When people say container they usually mean a bunch of kernel features (cgroups, namespaces, apparmor, etc) + an overlay filesystem.

I played with stuff like this since the chroot jail days and knew about containers before they were cool (anyone remember lxc?).

> This may be an unpopular opinion, but I’m not a big fan of containers and K8S.

It is unpopular for a reason.

Disclaimer: if you can run solely on cloud managed services + serveless, please do that and do not even look at the rest of this message. This is a very nice approach, although there are some things you need to setup before calling victory (deployment pipeline is one). And, as you mentioned, there is vendor lock-in.

Now, containers. Look, no-one WANTS containers. Or VMs. Or anything else. We just want to run our stuff. It just so happens that containers are one of the most useful abstractions there are. Unless someone else comes up with a new abstraction, containers it is.

Because a container is at the end of the day, a process. Are you against processes? Or against process isolation in general?

You cannot lift an existing service and run serveless, you need to modify it. In many cases, it is not practical. In other cases, they need to be an actual server-like application and hold a connection. Lambda doesn't help there.

> While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers.

One thing has nothing to do with the other. These are different levels of abstraction, there are challenges when running bare metal servers which are not present in cloud environments. Kubernetes can do so much more in a cloud environment (persistent volume claims, etc). Rolling out network attached storage on bare metal servers is a pain. It is also a pain with Openstack, but at least there is a standard interface there.

> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app.

Sorry, I respectfully disagree. I have spent the last two months implementing automation for deploying a cluster on AWS, with auto-scaling, auto-healing, the works, automatically deployed through Jenkins. It is NOT easy, it is not simple, and it is not focusing on my end application, unless you are ignoring all the technical debt you are incurring. And we DO have several k8s clusters, I will be moving that crap to k8s as soon as I can.

Let me make a quick list of what you need:

Prep:

You can use a barebones (ubuntu|redhat|coreos|etc) VM. In which case, provisioning is not complete once the VM is up by the ASG, you need to install the app. If you use an AMI, you now need to build automation to construct these AMIs. Note: if at this stage you are building AMIs by hand, this is a technical debt, which you will have to pay. Alternatively, you can use something like cloud init. If so, see below:

* Create the auto-scaling group * Create the launch configuration * If your AMI is not entirely complete, add user data (or equivalent) here * Set the health checks

And you are done! Right? No.

What about log rotation? Do you have centralized logging? No, tech debt. Go set it up. What about monitoring? Are you using cloudwatch? Prometheus? Go set that up. What about alerting? Not everything requires a VM to be destroyed, you need to set it up. What about upgrades? Are these cattle servers? Then you have to modify your AMI and launch config. Go automate this (tech debt if not) How are you controlling access? Do you have a team? Are they allowed to SSH? Where and how are you storing the keys? How do you invalidate if a key gets compromised? I could go on, but let's keep at this level because the point is to draw a comparison.

With K8s, here's what you do:

Create a container image. Dockerfile, fancy Jenkins script, some other mechanism, I don't care. Create an image, put it in a registry somewhere. Create a YAML file describing your 'deployment'. It can be a few lines of code if you don't care about most of the stuff. If you need external access, you can create a service, which is another YAML If you don't have an existing HTTPS load balancer, point to the k8s workers (trivial with something like ingress on GKE)

And you are done. This automatically gets you:

• Self healing • Scheduling among worker nodes. You can control it or let K8s decide • Bin-packing • Logging (centralized logging requires a one-time step, with fluentd or similar, may be handled by cloud providers) • Similarly, monitoring and alerting require a one-time investment in deploying something like Prometheus, after that is done. Getting prometheus to scrape your pods is very easy to do, easier than deploying in a VM by VM basis • Upgrades: deployments handle that for you. Even replica sets before it, you just needed to apply a new YAML with an updated version • There are no SSH keys to mess around. K8s has certificate-based user access control, with an optional RBAC • The SSH equivalent is kubectl exec • Service discovery: you have DNS records for all local services created for you. The cluster will direct you to the correct node. • Scaling is trivial, but most importantly, quick. kubectl scale deployment --replicas=X. It only takes whatever time is required to download the image and run it. You don't have to spin up a whole operating system • Optional: you can have horizontal pod auto-scaling, so your services can scale up and down automatically.

It is not perfect, but it can be a game changer. I cannot imagine how we would be running our operation without K8s. Actually I can: version 1.0 of the app was a bunch of VMs, one for each service. It was nightmarish. Now the push, company-wide, is to move everything to K8s. All VMs, all data stores, all of it. And it has absolutely nothing to do with hype, it has everything to do with proven advantages, compared to most of other alternatives.

I guess you could also do Mesos. They have a similar concept, only it's not K8s.

Note that SOMETHING needs to run the K8s cluster itself. That something is precisely your auto-scaling groups and VM images. It is less painful with a container-optimized OS (like CoreOS or whatever Google uses)

This is so true. I'm glad my team embraced containerization of applications internally, deployments before were a nightmare compared to being able to pulling the new image and running it running it or now updating the deployment to the target version of the application. When we need to debug a specific version, we can get the exact image of the application and its entire environment, not just checkout the revision that should be deployed.

There's something to be said though about turning entire productive development teams into chasers for cloud tech stuff. I've witnessed teams at startups who were quite focussed on their product to spend many sprints almost entirely on integration into k8s and other orchestration software. You could see minor k8s topics (like "minikube crashing on developer's box") popping up to slowly fill the entire backlog. I guess k8s just makes for an excellent excuse for procrastination. Then once you get your apps running, you need a whole staff of k8s experts to keep it running. It ends up being very, very expensive in terms of HR costs in my experience.

I'm getting k8s's goal to become a cross-cloud orchestration framework, and I'm as much of a standardization fan as could be. I just doubt the overreaching goals for k8s are worth it in the majority of cases, and have seen better business value in Mesos (though Marathon isn't where it could be) because you can realistically run it on your own premises.

I guess k8s is Google's vision for a self-service cloud platform that offloads everything to configuration details on a uniform matrix of nodes, and in particular such that Google doesn't need to provide customer support. I just don't see the benefit for the customer, considering we've been running POSIX workloads for almost 50 years now.

Thank you! Every time k8s is mentioned we get comments like the OP: misinformed get off my lawn/hacky bash scripts/just give the infra people a zip file to do the needful, because containers are wrong and k8s is alpha at best opinion. And it hurts to see people be willfuly ignorant and be without physical keyboard, and it also is tiring to try and show them they're wrong. Thank you for this comment. You did good.

thank you for writing this up.

> I have spent the last two months implementing automation for deploying a cluster on AWS, with auto-scaling, auto-healing, the works, automatically deployed through Jenkins. It is NOT easy, it is not simple, and it is not focusing on my end application, unless you are ignoring all the technical debt you are incurring.

yes, i can appreciate that having a system automatically handle even some of this necessary plumbing in a reasonable and standardised way is attractive.

A container is not a process. The fact that you say this makes me wonder if you understand what K8s and how it works.

K8s is not going to solve the issues you outline above (logs, proper monitoring, etc). Even worse, you’re gonna have a bad time migrating them to a proper solution.

While k8s does not in and of itself solve some of the issues pointed out above, it does centralize a lot of these problems. These centralized problems are then often easily addressed by the cloud providers actually running the k8 cluster. GKE automatically sends log data and metrics through fluentd to their Stackdriver platform including error alerting. If something even prints something that looks like a stack trace to stdout or stderr, Stackdriver sends alerts and creates an issue for us.

To the first degree, it is. The way people use containers is typically one binary starting up, which equates to one process in Linux. Sure, docker et al have overhead, but that's constant overhead, and much of the hard work is done in cgroups.

You just might be wrong here, friend.

> If your app needs a container to run properly, it’s already a mess.

> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense

So running with container isolation is a mess, but serverless isolation is admirable?