If you don't care about a centralized API to probe status and manage each instance, Ansible should be enough to orchestrate these installations and with little effort (that also depends on the application at hand) getting zero-downtime rollouts with Docker can be easily done with it.
However, if you want a single control place to probe status and want to avoid writing your own rollout scripts, Hashicorp's Nomad [1] might be a good solution for this. It is a lot simpler than Kubernetes while still giving you nice primitives to describe jobs/services, health checks, rollouts strategies and etc. Treat every site as a datacenter of its own, setup a job of type "system" (akin to Kubernetes DaemonSets) and all you need on these sites is internet access to your HTTPS endpoint of the Nomad control plane.
If you want to talk more about this, hit me up on Twitter or Telegram, I use @rochacon as my handle virtually everywhere.
[1] https://www.nomadproject.io
Edit: grammar and typos
It's a bit overlooked now because every DevOps person nowadays seem to think Kubernetes is the only rational thing as it will look good on a CV.
I predict Nomad will be on the upswing the next few years as people realize Kubernetes is extremely complicated to self host.
I had to write my own custom blue/green deploy script to hot reload traffic to proxy_pass definitions in nginx upstream configs since I don’t use Docker.
Since docker swarm knows if an instance is down, it will know to not use that instance.
If you need a practical suggestion use simple solution like LXD [1]. But as you are already using docker stay with it and ansible.
As you are using ansible you are already doing infrastructure as a code, you are way ahead of curve already.
Probably when you grow a bit bigger use packer [2] and terraform [3], but I think ansible will do just fine.
Kubernetes was designed for Google kind of problem the burden to maintain is quite high with lots of moving parts, unless you use Google GKE or Amazon or managed kubernetes service. So don't fall for it unless you need it for your resume.
[1] https://linuxcontainers.org/
> Digital Ocean now offers managed Kubernetes for free
GKE is just as "free" as DO, in that you only pay for the nodes you actually use (those nodes are probably a bit more expensive, admittedly). EKS is the outlier in this trio in that you also have to pay (a lot) for the master. Could you clarify what GKE could do to "follow suit"?
If you have a single node not a cluster in each location then k8s buys you a lot less. Some people still advocate single-node k8s "clusters" for benefits like rolling deployment.
I only have limited experience, but Kubernetes works very hard to abstract away the actual machines, and it works best that way: you just say "Deploy 20 instances of job X", and k8s will somehow find the machines to deploy them. You don't care where they are running - k8s handles that.
Once you start to care about actual machines and which jobs are running where, k8s starts to make less sense. You're paying for huge complexity (required to abstract away the machines), but you're getting none of the benefits - it just becomes a glorified wrapper around Docker daemon.
We run on multiple clouds, so vendor apis and k8s implementations are not usefull to us.
If you have a 100 user SaaS platform, do you run Linux containers on top of Linux just for isolation?
Docker is useful when you don't want to care about the underlying operating system or you might be deploying on different operating systems etc. Also if you want to quickly switch providers (like going from Ubuntu 18.04 on digital ocean to Windows Server on Azure) since it's a self contained image.
This self contained image is useful during development too. I can cleanly separate dependencies between applications and dev/test/prod. For example I develop a node app based on node 12 + pg 11 but I tinker with ghost too which requires node 10 + mysql. Having this in Docker keeps cruft and complexity off my macOS installation at the cost of space. Having multiple versions of a runtime/database etc can quickly become a nightmare to manage on local operating systems.
Operationally if you can control deployment and you don't need many servers, deploying to the base operating system makes sense. Docker/Kubernetes shine when you have scale + need to provision many servers across different platform providers.
On that same note... when do Terraform / Ansible start to make sense?
in dev: You can setup dependencies without worrying about underlying os. So it is easier for building a new dev machine, adding a new team member, etc.
in integration test: you can setup multiple integration test environment as long as you have enough resource. Running multiple integration tests without interfering other team members is the major benefit here.
in prod: as you may know, for isolation and horizontal scaling. Easier roll back and update?
User-to-user isolation is hard, and extremely hard if any of the users could be malicious. Docker is broadly neutral here - adds a few tools, might add a few vulnerabilities.
I'm really upset with the way Google let their marketing team run roughshod all over the place with that software. Kubernetes is almost never the tool to use. It's entirely insecure, overly complicated and almost never fits the intended supposed benefit. Worse it feels that the entire CNCF ecosystem is ran entirely by marketing people with "developer evangelists" that have never coded a single line of code in their day - it's a real shame and quite honestly an insult to professional engineers.
Ansible/Puppet/Chef/Salt: Then you use this to install your stuff in the VMs. Just pick one of these and stick with it.
K8s is not about configuration management, it’s about dynamic application management. Some parts infringe on areas where CM tools work as well, but k8s is all about managing containerized applications.
Trying to set up flows for ”works on my laptop”-dev, ci/cd, loadtesting, a/b tests, canary releases, autoscaling and rollbacks for multiple teams of devs? K8s really simplifies these things.
The idea is that you have one api spec to rule the whole stack (from a dev perspective). If you go down a more light-weight stack, a lot can still be achieved. More duct tape required though. That being said- I love duct tape!
https://docs.ansible.com/ansible/latest/modules/list_of_clou...
Those are tools for configuration management. If you instead use packer, docker, you can build your vm/image at build time, and use Terraform to setup vm with that image. Use etcd (in the image set to pull config) or similar key-value for distributing configuration.
Not "setup base vm with terraform" and then "ansible to install and configure it". Just build your vm/container image with the software you want already installed, and a etcd or other pull-configuration from a pre-set source. Done.
Now you dont need configuration-management, and dynamically changing infrastructure, since you moved it to the build-step.
I’ve never seen this combo of requirements on any service before, what is it? Or are my assumptions just wrong about what these things suggest about your business?
Ok, joking aside, intruiging as there seems to be a few companies providing monitoring services for various things. How did you all find the market for this?
Obviously you don't have to spill the company sauce but I'm trying to imagine the benefits of having all of this data.
So hmmmm, a few things to think about for the tooling:
() How quickly can you notice a single-node service outage and react / restore service?
() How much downtime is ok? Is it ever ok to fail to process data for an entire day for one of the ~50 markets (can you just say "this report is incomplete" and reprocess it later)? How much wiggle room do you have (how long does it take 1 instance of the application to do its local processing job on 24 hours of data, and how long does the downline data pipeline have?)
() Are you responsible for the part that watches TV + saves it to disk, and ALSO the part that batch-processes the saved video and produces the ad metadata / quality metadata? Are those different services deployed separately?
() Yikes, is there, like, a person on call in each of 50 places who can unplug the thing that's watching TV and plug it back in, if a wire burns out or if the machine that's watching TV fails? That sounds like it could be an operational nightmare but maybe this is a solved problem / maybe you can buy watch-TV-and-save-it-to-disk as a service?
() How do you make sure that every node is running the same version of the software with the same version of all the configuration?
() How do you release new versions of the software? Do you (a) have a non-production environment which runs side-by-side at every node, processing the same data, and measure that the output is equally correct or better, the performance is the same or better, and there are no new errors or failures -- then promote the non-production code to production automatically after a while? Or do you (b) deploy a "canary" to one of the 50 nodes and watch how it performs for a while, then deploy to the rest if they all behave okay? Or do you (c) just ship the latest code to all 50 nodes on Friday nights when no one's looking, and check some health metrics over the weekend, and roll it back on Monday if a metric looks bad?
() How do you keep track of what software versions you released when? If someone asks "What changed on 'X date'" does your tooling let you tell them pretty quickly and pretty accurately?
() If you discover a bad bug (the parser is corrupting data; the new parsing job has a slow memory leak and fails catastrophically after 21 days of uptime) and you need to make a change in a hurry, do your runbooks let you make that kind of change quickly?
Doesn't seem like you really need k8s or ansible or chef or whatever for this, as much as you need to write down your operational requirements and decide what technology will meet them.
That said, maintaining a large cluster is complex too. Often, there is much more involved than making sure each image's config files are up to date. Kubernetes shines when you have a number of different docker images, where each needs its own unique network, scaling, cpu, and disk settings, i.e., hardware requirements that can't really be controlled with just config files.
In your case, the fact that all of your images are identical probably cuts against using Kubernetes. However, the fact that these images are running on different networks where each may require their own tweaking leans towards it. If you expect to start adding many different docker images to your cluster, with different hardware requirements, it's probably worth the learning curve to get on to Kubernetes.
Disclaimer: I've never run Kubernetes on bare metal, just hosted environments. Bare metal may make Kubernetes much less attractive.
* A unified control plane (monitoring, statusing)
* An automated rolling-upgrade system
* "self-healing" in the form of automatically restarting your containers if they go awry
However, for your use-case, it is likely more effort than it's worth:
* Settin up and maintaining Kubernetes on bare metal servers is a huge amount of work
* A lot of Kubernetes features are not needed for your use-case
* The Kubernetes networking model essentially requires you to set up private network spanning all of your sites in order to work but provides almost no benefit for your use-case
Applications should just be a `Deployment` manifest with the type of distribution being a child setting (X replicas with affinity or 1-per-host). There's no reason to have competing ReplicaSets and DaemonSets just for this detail.
Win with k8s is that once you’re tuned, a lot of “config crap” is abstracted away, basically. But - there's a reason ansible is supported specifically through the “operators” SDK, something we use to manage seamless upgrades of a k8s deployed app:
https://github.com/operator-framework/operator-sdk/blob/mast...
Edit - forgot that if meteics based autoscaling of the app is a requirement my experience is that k8s makes this a breeze. Probably not a req if using single node hosts though...
I switched to saltstack some time back and it's much better while having a similar sorta interface. The docs are worse than ansible imo tho. ymmv
See ControlPersist/ControlPath OpenSSH client configuration.
It's not meant to have the connections persist forever, just long enough for repeated short ssh connections to have almost no overhead, and therefore be much faster.
No, you don't need Kubernetes (though fwiw if you used it what you'd have would be 50 nodes with a single daemonset, 1 pod of the app per node, if that helps understand) - but I'd suggest not using Ansible.
Instead, use Packer (or similar) to create a machine image/snapshot/whatever your server provider calls it, and then deploy that same image on all 50.
You might have some small amount of host-specific things left to configure, and by all means use ansible for that if you want, but there's no need for error-prone running of the same large playbook 50 times.
Personally, I find that final setup is little enough to do it with Terraform (which is provisioning the servers anyway).
I'll soon have to manage 50 remote bare-metal servers, all of them in different cities. Each one of them will run the same dockerized application. It's not 1 application distributed in 50 servers; it's 50 instances of the same application running in 50 servers, independently.
A frequent use case is to deploy a new version of this application on every server (nowadays I do it manually, it's OK since I manage only like 10 servers).
A nice-to-have would be to avoid downtimes also, when I update the application (nowadays I have a downtime of 2-5 minutes (when it goes well), which matters for us).