Pods die, Kubernetes spawns new ones. Server goes full, Kubernetes spawns new node. Server runs out of disk space, Kubernetes evicts nodes until everything is good again. Server goes down, Kubernetes shuffles all the pods over to a different node. And so on. It does this at a scale of hundreds or thousands of machines. The utility it brings also applies to small clusters. You don't remove ops, you just flatten it into Kubernetes' operational model.
Equally importantly, Kubernetes also brings a declarative deployment model. You have the same with Systemd, but at the node level. Kubernetes is a bit like Systemd at the cluster level; its "units" are pods declared in YAML/JSON files, but unlike Systemd they run on any node the cluster, and are subject to a resource allocation model that includes things like persistent storage (e.g. AWS EBS volumes). You don't get anything close to that with Systemd, which is much more analogous to Docker.
I haven't heard anyone ever claim that Docker brings reproducible builds -- why would anyone assume that? What Docker give you is reproducible deploys; build once, deploy many times from the exact same image. Maybe that's what you're thinking of?
But, for the sake of the argument:
> Pods die, Kubernetes spawns new ones.
AWS ASGs also solve for this, as does every other cloud provider at the infrastructure level. For applications, systemd and any init system can restart an application when it crashes...
> Server goes full, Kubernetes spawns new node.
I assuming you mean utilization - again, ASGs/CloudWatch solve for this as do every other cloud provider (if they didn't a few minutes worth of Sensu plugins and good-to-go).
> Server runs out of disk space, Kubernetes evicts nodes until everything is good again
Right but this can and has lead to disaster. I got paged for bad response time and found out the issue was that containers were being evicted and pods were going critical because _all_ hosts were filling up on diskspace and there were no healthy nodes on which to put pods. This due to one bad service writing out tons and tons of log data). Had that been dedicated, I would have just gotten paged for one full disk and fixed the problem when it showed up instead of waiting for a bigger issue. This is super fixable in Kube (and was - that service now has its own volume and still lives in kube) - but my point is that it's a problem for everyone.
> Server goes down, Kubernetes shuffles all the pods over to a different node.
This effects response time of user facing applications as new 1GiB+ containers are shuffled around (I know, thats an unfair example). ASGs and Cloudwatch solve for this in a saner / safer way.
As for the reproducible builds, it was in response to the article which says:
> The ‘repeatable builds’ argument is the most interesting to me, in part because this is the same promise that java made in the 90s (write once run anywhere).
You may be correct that Docker doesn't promise this explicitly, but it is a big misconception. Additionally, `tar` solves for "reproducible deploys"...
I'm not an AWS employee, just figured I'd use a common cloud as an example.
- the deployment artifact is a "fresh" instance that's "booted" in a fraction of the time—as in a couple of seconds vs a few minutes—than a new EC2 instance would be
- since the deployment artifact is a fresh instance, there is no need to manipulate an existing instance already running your application as part of your deploy process (e.g. scp, ssh, unpack, run commands, restart/reload application)
- if you design your build pipeline appropriately, the actual data needed to be shipped is effectively just a diff from the previous image. In my org, that could mean the difference between shipping a 300MB vehicle to ~300 instances and shipping a 300KB vehicle to almost certainly less than 300 Kubernetes nodes
There are other things as well, but I wanted to stay roughly within the parameters of your examples.
If your organization doesn't already have a continuous integration story, I'd say it probably isn't ready for containers in production. If your cloud footprint isn't costing you let's say >= $50k/mo or so, I might argue that the engineering effort to Kubernetize your things is probably better spent in more direct ways on your product.
If you do have a CI/CD story, and you do have a very wide, expansive, and expensive cloud footprint, containers and Kubernetes are extremely compelling possible solutions.
I'm in that situation now (with a SaaS product), and my approach so far has been:
* set up tests
* set up Docker images
* set up docker-compose for easy local deployments
* do lots of work to make the app behave inside Docker
* take control of AWS and IAM using CloudFormation, making it easy to set up and roll back permissions
* bootstrap KOPS/K8s with CloudFormation
Future work:
* set up Jenkins CI on top of K8s
* K8s deployments of the app for CI
* staging K8s deployments
* K8s in production
This has been my first venture in operations and it's been very interesting and rewarding, but at times it gets very intimidating to "be" operations with no senior guidance. I enjoy the freedom and responsibility and I feel like I at least have a significantly better grasp on this than anyone else in the company, but I wish I knew of a way to feel like my work was good in an absolute sense, like "definitely good enough".
Has anyone here gone through a similar scenario, bootstrapping ops infrastructure in a small business? Would you have any remarks regarding architecture, or the politics required to make the importance of solid ops and the improvement of software reliability and security clear?
What's different about what you're referring to above and what the commenter mentioned? It seems to me like they were designing the app to be able to scale.
Say they are making a crud app. Keep no state in session. Deploy as easily as possible. Boom, done. Later you can add a load balancer to scale to 100 servers. Right now you can run it on 1. It is 'designed to scale'. But actually running your single app in a 1 node k8s cluster? That is days of extra work for nothing right now.
Yes, of course you may be working in a startup that requires 25GPUs to serve even one customer. No sarcasm. I can imagine some startups that might meet that requirement.
But there are an awful lot of startups that start massively overengineering their footprint early when all they need is a web server with a cold or hot spare (use a load balancer with automatic failover if you've got one in your cloud or something, if not I wouldn't stress, and automatic failover on prototype-level application code can cause its own issues) and a database server with a good backup story and some mechanism for quickly bringing up a new one if necessary. (This generally leads you to some sort of clustering thing or a hot replicated spare because it doesn't take long before your database requires hours to rebuild from a backup or something.)
You're often better served just giving occasional thought to how you might split things up in the future and using that to at least hint your design than actually splitting things up immediately.
The pros to Docker so far: Dependencies: Dockerfile gives a list of explicit system dependencies for each app. This can be done in other ways with package files or config management but this was not being done before and this is an easy catch all to force it for any different type of environment. Logical Grouping: App environment (Dockerfile + docker-compose.yml) lives alongside codebase in a single git repo Deployment: Deploy to any box with `git clone myapp && docker-compose up` for testing/dev instances or migrations Development: We mount the codebase from a host directory into each container, with git hooks to update the codebase, which works well for us (we have no CI) Plus it's fun!
Cons: Operational Complexity: Devs/Ops teams probably won't want to learn a new tool. I setup a Rancher instance to provide a GUI which makes things a bit easier to swallow. It has things like a drop in shell, log viewer, performance metrics, etc. Network complexity: we never needed reverse proxies before, now we do. Clustering/Orchestration: We don't cluster our containers, but the more we add the more I think we might want to, which would add a whole new layer of complexity to the mix and seems unnecessary for such a small shop. Security?: lots of unknowns, lack of persistence can be bad for forensics, etc. Newness: Documentation isn't great, versions change fast, online resources may be outdated.
Like you, I'm sometimes unsure if this is the right choice. Maybe a monolithic server or traditional VMs + Puppet would be easier, simpler, better? In the end, I think Docker just fit with the way I conceptualized my problem so I went for it. You may never get that "definitely good enough" feeling, but if it fits your workflow and keeps your pipeline organized and manageable, then I say go for it.
In my personal opinion, this is all good work to do. One minor nit: You probably want to get out of the habit of thinking about `docker-compose up` as a deployment, and just think of it as a much cleaner, much more "prod-like" local development environment.
> * do lots of work to make the app behave inside Docker
I'm curious about this. What was the overall nature of the work? Things you would have had to figure out in a distributed platform anyway, or container-specific issues?
> * take control of AWS and IAM using CloudFormation, making it easy to set up and roll back permissions
This is amazing, I applaud you for your foresight. I wish we had known about this pattern much earlier.
> * bootstrap KOPS/K8s with CloudFormation
This is where there's going to be another giant time sink, unless you're already pretty familiar with Kubernetes' resources—Deployments, Services, PersistentVolumeClaims, Secrets, etc—how they work together, what they enable, what their limitations are, and so on. Not to mention picking a network overlay plugin, deciding on a service discovery model, and understanding how Kubernetes works. BTW I am currently working on this very knowledge, and at least for me it's been pretty heady stuff at times.
>Future work: >* set up Jenkins CI on top of K8s >* K8s deployments of the app for CI >* staging K8s deployments >* K8s in production
Everything here seems relatively sane, in broad terms. I mainly caution against moving too quickly and also against spending engineering efforts on this if you are also an IC for the applications your teams are writing.
>This has been my first venture in operations and it's been very interesting and rewarding, but at times it gets very intimidating to "be" operations with no senior guidance.
Honestly, you seem to have the right mindset for this and so far you haven't mentioned anything that I see as a huge red flag.
>I enjoy the freedom and responsibility and I feel like I at least have a significantly better grasp on this than anyone else in the company, but I wish I knew of a way to feel like my work was good in an absolute sense, like "definitely good enough".
Welcome to ops! There will never not be an operational issue that either needs to be fixed or is silently waiting to strike at 3AM on Monday morning.
We use docker to solve the distribution/deployment headaches where we don't know and definitely cannot influence the (different) environments where the package will be deployed. Docker made a headache go away for us - we can make a deployment available to others, they can run it, and it generally will just work properly the first time.
The earlier alternative for that style of "reproducible deploys" was full virtual machine images, not tar. The alternative in the years before shipping VMs was easy was configuring it manually on site or (if it made financial sense) shipping pre-configured hardware. Something like tar was never a reasonable alternative for deploys - even decades ago when big unix vendors controlled/owned/managed your whole system, them 'simply deploying' a tar package worked usually, but didn't work reliably.
Additionally, one big complexity re: Kube on dedicated is dealing with custom hardware. Kube and Docker really require modern and specialized kernels - This actually applies to AWS as well; the current Kube AMI for AWS doesn't install the `linux-aws` kernel for example, so you have no access to the ENA driver or the ixgbefv driver, and the t* and i3 instance performance is awful (the most common and the most vital machines respectively). There is a PR for that open somewhere and I'm sure it will be fixed soon, but you see my point.
After the hardware/kernel build (which you'll need to manage outside of Kube, so you'll probably choose something like Ansible), you'll probably want auth management at the host level (assuming you run on-prem), which you'll most likely fall back to Ansible for again.
I should probably also admit somewhere in this thread I actually use and prefer Chef :)
Unfortunately, the same culture which fucked modern web design (a new JS framework every 6 months) is starting to creep into Ops; some REALLY smart people make a tool which promises to be X but with X's main problem solved. They do the easy 80% and then abandon the project when the hard 20% needs to be fixed (and forget documentation).
That's a nice summary. The 2nd level ignorance of those people is astounding.
Oh, sure, write the 618th package manager, I'm sure you'll be the one who finally gets it right.
No need for JS for what can be done in plain HTML/CSS.
It can also potentially help with things like caching, but more traditional server-side templated designs can also use things like Varnish with the ESI feature, utilizing caching mechanisms in the framework / app (e.g. Memcache or whatever), etc.
I used to think SPAs were dumb, but now I love Rails 5 API-only mode + EmberJS + EmberData. It also makes server-to-server a breeze, because you already have your API. Mix in JSONAPI, swagger, and oauth and it's a pretty happy environment to build software in.
c2.com being a tragic example of this.
I was wrong. Load time was an abysmal 14 second(!) wait. I normally don't make jokes on HN but today I'll be pouring out a sip of dark roast for my metaphorical 'dead homie'.
Offloading everything like that to client is antisocial behaviour. Seriously. This goes beyond "bad engineering"; for any popular website, this literally means saving a bit on your servers while causing lots of electricity being wasted by your millions of visitors. This is the computing equivalent of selling extremely fuel-inefficient cars to regular people.
And I don't buy the argument that javascript-based client apps scale better than servers rendered partials where there's been over a decade worth of solutions for scaling out the server layer. Also, you'd be applying a lot of the same techniques for scaling a pure API as something spitting out partial web views.
You can achieve that with a little sprinkle from jQuery.
For UI/UX rich experiences, taking advantage of OS animation frameworks and widgets, I would anyway advise native applications.
wat.
> terrible "wilderness years" of server templates and jQuery
where things go off the rails.
> terrible "wilderness years" of ... jQuery
is the part where things go off the rails (and literally off the Rails).
https://developers.redhat.com/blog/2017/05/04/breaking-up-th...
https://en.wikipedia.org/wiki/Nitpicking
I did not mean to imply your comment has lice :)
My guess is this technology will be some kind of cluster management. If there were a 'platform' for horizontally scaling data that managed sharding, rebalancing & backups plus gave me user-space hooks to override policies, databases might slowly & cautiously build on top of it.
Most DBs have their own of these (pgpool, mongos, redis sentinel, all of cassandra & hadoop) but would be cool to centralize the common reqs and collaborate.
As far as databases and horizontal scaling, this could probably be done currently within each Docker container at the database layer as you mentioned (CockroachDB also comes to mind). So let's say you add a new docker container with its own persistent+redundant storage volumes, and that container is running a cockroachDB node. You can add more containers as cockroachDB nodes, each with their own persistent+redundant storage, and you get all the DB sharding etc across nodes from cockroachDB while the redundancy+persistence are handled by Docker (plus storage solution) as far as restarting, migrating, reconnecting, etc. This probably has all sorts of problems I'm blind to but that's my rudimentary understanding.
Handling horizontal scaling, sharding etc at the storage layer directly through Docker sounds much much harder, and I don't even know what I've been talking about so far so I'll leave it at that
Here's a list of some people doing persistent Docker storage :
Flocker
Portworx https://portworx.com/
StorageOS https://storageos.com/
Rancher Longhorn https://github.com/rancher/longhorn
Quobyte https://www.quobyte.com/containers
Nexenta https://nexenta.com/solutions/containers
Infinit https://infinit.sh/docker
EMC REX-ray https://github.com/codedellemc/rexray
Ceph
Gluster
Mesos https://mesosphere.github.io/marathon/docs/external-volumes.html
Kubernetes https://kubernetes.io/docs/concepts/storage/persistent-volumes/
vSAN
Nutanix http://next.nutanix.com/t5/Nutanix-Connect-Blog/Containers-Enter-the-Acropolis-Data-Fabric/ba-p/11122
Netapp (Docker volume plugins for SolidFire+ONTAP+E-series)
AWS EBS/EFSOh baby the desire to pitch my WIP SaaS product. This is something I'm extremely keen on and I couldn't agree more!
docker run .... -v /storage:/storage myapp
That's not what people are talking about. If you run a container on smartos and the physical machine it is running on blows up, what happens to your persistent storage?- starting the various pieces of the app in the correct order - permissions (uid, cgroup, maybe even virtual file system) - environment variables, working directory - passing around sockets - restarting things if needed - monitoring resource usage - centralized logging
The idea is that Linux already solves that vast majority of these problems, as do cloud providers. I then ranted on a couple of the "problems" that containers and container orchestration "solve".
You're right tho - a bit of a ramble. Been a long day / on-call week :P I will think about how I could have been more eloquent for next time.
Deployment tool, not configuration tool. Ansible is awful for keeping environment configured, it was from ground up designed as somebody's deployment script.
Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers. I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management.
I could easily imagine that the 'modules' in Ansible aren't as robust as those in, e.g. Chef, at least for some platforms. But my experience so far having used both Chef and Ansible with Windows VMs (on VMWare) is that Ansible is much better.
Hosts that were temporarily down when Ansible tried to ship configuration. CFEngine or Puppet just apply the changes some time later without operator's intervention.
> Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers.
So does Ant or Gradle. Something is different about the operation, though, because nobody uses Gradle to manage server configuration.
> I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management.
The architecture and the protocol.
With Ansible you get synchronous sequence of operations that breaks on first problem (e.g. host is down for maintenance). This is OK for deployment, but not for ordering a configuration change, for which you don't care when it will be applied, as long as it's soon enough. Note that deployment and managing configuration are similar, but different things, and it's not uncommon to need to do both (although not at the same time).
Then the protocol. Ansible uses brain-dead idea of using the debugging/interactive channel (SSH) to configure things, including possibly SSH itself. One small mistake can cut off your all remote access at once. It's negligible risk if you only configure SSH daemon when deploying the server, but it increases quickly in the log-term.
Not to mention that managing SSH access properly (e.g. not ignoring warnings about hosts' public keys, but not just that) is much more tedious than keeping a CA, so SSH is a bad choice for unattended operations.
Ansible is great for fire and forget machines you don't need or care about. You need real config management with an audit trail when you go enterprise.
I really disliked SystemD before I got my hands dirty with it. Assuming you're developing with any modern language, the language itself probably wraps OS differences anyways (Node / Golang / Rust / Ruby / Python / anything-libUV-based to name a few), not like you can convince developers to change their habits anyways.
People act like Docker makes builds reproducible by magic, then go on to not pin any Golang deps or curl internet resources or not generate lockfiles for NPM, ad infinitum.
There is and never will be a shield operations people can put around their developers. You just need good developers. If you want SoA scaling or automatic cloud configuration, why not just use the most established tools available? Linux's init system and kernel _already do_ resource management, even outside of LXC. People act like a "cloud scheduler" is the only thing that lets multiple processes work together.
This whole idea of isolation is also really inane. Each app will have different disk, io, network, cpu requirements. I also dream of the day Kubernetes can do this "sorting" better than an experienced operations team can, but that day is _far off_ (and I will die before I put redis/mysql/postgres/cassandra/any-other-mature-datastore behind docker and iptables)