> Complexity has high costs
Complexity management is the central theme of building any large, valuable system. We would probably find that the more complex (and correct) a system, the more valuable it becomes on a relative basis to other competing solutions. The US tax code is a pretty damn good example of complexity intentionally taken to the extreme (for purposes of total market capture). We shouldn't be surprised to find other technology vendors framing problems & marketing their wares under similar pretenses.
The best way to deal with complexity is to eliminate it or the conditions under which it must exist. For example, we made the engineering & product choice that says we do not ever intend to scale an instance of our application beyond the capabilities of a single server. Consider the implications of this constraint when reviewing how many engineers we actually need to hire, or if Kubernetes even makes sense.
I think one of the biggest failings in software development is a lack of respect for the nature and impact of complexity. If we are serious about reducing or eliminating modes of complexity, we have to be willing to dig really deep and consider dramatic changes to the ways in which we architect these systems.
I know its been posted to death on HN over the last ~48 hours, but Out of the Tar Pit is the best survey of complexity that I have seen in my career so far:
They’re always asking if you’ve worked with the latest tech stacks, never if you can set up a really well optimized nginx server.
I worked for one startup lead by a Java Architecture Astronaut, and working with the byzantine patterns and build systems made adding even simple features a morale-draining slog. It killed the product.
It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box. A couple of problems with that:
1. What happens when you have to reboot to apply a kernel update?
2. The geographic location of that single box is itself a gap in redundancy. This is one thing I like about AWS and the other hyperscalers, with their regions that each have multiple data centers connected by a private network, with load balancers and other things spanning the region.
My conclusions so far are this:
Single node application systems are by far the most reliable and manageable from a business logic standpoint. At no point does spreading a problem across more than 1 computer make that problem easier to solve.
If you are concerned about latency, you need to get really abstract with your problem and ask what is even possible in information theoretic terms. If you are truly constrained to 1 serialized, synchronous context (i.e. a competitive counterstrike match or a stock exchange), there is little you can do to alleviate the root problem as your users get further from the server. You can certainly look at using some consensus protocol like multi-paxos, but then your transaction latency goes from microseconds (if you were clever) to milliseconds, representing orders of magnitude slowdown in the typical case.
The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system.
For the customers I work with, it is wayyy easier to build & sell a system that operates on a single box with sync replication + manual failover. Our customers are tolerant to production having a brief outage for a few minutes during the business day. Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable.
Manageable? Maybe. Reliable? No. Most companies don't need it but if a pipe bursts above the server room and now all of your medical records for your hospital are unavailable you're going to have a bad time.
> At no point does spreading a problem across more than 1 computer make that problem easier to solve.
I don't think anyone claimed it was easier (yet). The main thing people strive for is:
1. Dynamically reshaping your compute needs to match any needs.
2. Surviving failures of nodes or crashes in your application.
3. Managing all of the BS that goes with obtaining this (logs, etc).
> Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable.
I hope this isn't the case but this sounds like quite the death flag.
That's not actually something to brag about, as it implies they weren't receiving important kernel updates, particularly security updates.
That said. We communicate directly with other systems on the same premises that probably haven't been rebooted in over a decade.
Building something that can run 10 years without downtime is not happenstance. It is very intentional and deliberate engineering.
Allows you to patch the kernel without reboot.
> The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system.
This part was what I felt deserved a counter-point though. Consensus is indeed at the core of the issue once you want distributed fault tolerance. However, I think you'll quickly hit two things with your approach: 1-2ms of latency, I fear that it may come with highly correlated failures on the "first tier of resilience". Moreover, the "second tier" being much farther, keeping them in consensus will imply harsh trade-offs. If you use synchronous consensus protocols, you'll slow down drastically the "first tier" (assuming you want consistency), if you go for the asynchronous replication (not consensus, this matters...) then the second tier can't really intervene on leader election or failover without risking a partition on a false positive (and if you try to be conservative there your RPO will suffer).
If you're into these (fundamental) issues, I'd recommend Leslie Lamport's work (i.e. https://www.college-de-france.fr/site/en-martin-abadi/semina... or http://www.lamport.org/), the paper pointing a disappointing impossibility: https://dl.acm.org/doi/abs/10.1145/3149.214121 and its generalization (https://dl.acm.org/doi/abs/10.1145/167088.167119).
Where your datacenters are geographically located is usually a big first step in even starting these types of conversations. The nature of "maybe sync replication represents a liability or is feasible" might be a conversation about the geography of a region and statistical likelihood of certain disasters impacting multiple sites simultaneously.
Some customers cant ever afford to lose a single transaction no matter what, some just need it to be reasonably stable but incredibly fast (e.g. gaming vs banking).
Will definitely be spending some time reviewing Lamport's works again. Establishing the notion of stable time between all participants is a fascinating way to solve a lot of problems in distributed systems.
If needed, I can choose to apply the update in the middle of the night in the timezone of most of my visitors.
If it really is a sensitive app and I can't afford any uptime, I just add another inexpensive box, put a load balancer in front (a Cloudflare load balancer would work fine). And since I now have 2 servers, I need a way to manage them without having to manually log in to each of them each time. Enter Ansible. And that's it.
2. Now that I have two cheap boxes, nothing prevents me from having them in two separate data centers and two separate providers.
This is what I have. 2 boxes only. One on my own premises in Canada and another on Hetzner in Europe. Each one can go down, no sweat. Also I do not strive for "live uninterrupted updates". All my applications and processes allow for short disruptions not to interfere with the main course.
Also I write native servers in C++ for backend and they process thousands of requests per sec without breaking much sweat. Way more than my business would ever need.
Same situations with my clients as well with the difference that for legal reason they rent computers or whatever goes as such from Amazon or Azure.
2. The latency within AWS is easily as high as US-EU backbone latency. Also, most web apps these days need 2000ms to load all the tracking and advertising crap, so 100ms location latency are negligible in comparison. Plus for most real companies, you'll have one website per country anyway. One server for .com and one server for .eu
No one should or is using Kubernetes to run a simple LAMP stack.
But if you have dozens of containers and want them to be manager in a consistent, secure, observable and maintainable way then Kubernetes is going to be a better solution than anything you build yourself.
Yes they are. Some developer got all excited about the capabilities of k8s, and had an initial larger scope for a project, so they set it up with GKE or EKS, and it managed to provide just enough business value to barrow in like a tick that won't be going away for years.
Developers get all excited for new shiny tools and chuck it into production all the time, particularly at smaller orgs.
1. Start with basic LAMP app in git that's manually deployed to an EC2 instance 2. Add in CI / CD + CodeDeploy 3. Create a staging environment 4. Dockerize local environment to keep dev environments in sync and onboard easier (really, this part's a gamechanger for a small company) 5. Ok so now we have Docker for local dev environments but stage and prod are managed separately. Can we just run our Docker containers in stage / production?
When I researched step 5, the options were basically k8s or Docker swarm but Docker swarm didn't seem battle tested (for prod). k8s was clearly a nightmare for a small team to maintain so we started looking into GKE / EKS -- but EKS was still in beta. Thus we punted. We've actually started using ECS for a newer project and I'd likely go that route for step 5 instead.
So they weren't trying to run a simple LAMP stack then.
I recently asked a friend who still works there if anything else has been pushed to k8s since I left (6 months ago). The answer: no.
Au contraire! This is very common. Probably some combo of resume-driven devops and "new shiny" excitement.
i started by simple wrapper around docker 8 or so years ago. over the years we’ve moved to k8s b/c it provided essentially the same api we had home grown. this reduced the LoC and moved the complexity of something like dynamic reverse proxy via nginx downstream into kubernetes ingress abstraction backed by nginx-ingress-controller.
Two groups of people shouting past each other.
And 99% (rhetoric number of course) of needed solutions would be served by a single real server (rent one on Hetzner or wherever) with 0 need to ever upgrade.
But sometimes with the big microservice/kubernetes solution, the configuration space is so big that miscommunications/mistakes could potentially take you down for more hours/year than the downsides of being single-hosted would. So now you invested all of that effort and for what?
One company I worked for wrote a distributed scheduled jobs system that was immune to single-machine failures, it went down like 4 times in a year, messing up my team each time. I was like "guys, if we just provisioned one machine with cron and no failover, it would have better reliability".
Besides if it warm the cockles of your heart so much you can always take any of my products and shove it in the container and run it under k8.
Or, to channel Fred Brooks, essential and inessential complexity.
Which items fall in this category?
Yes, it is complex, but there are lots of use cases where it is the most elegant and least complex solution. Yes, it definitely does not make sense to use it for a lamp stack deployed to one server, but there are use cases where it's a huge improvement (e.g. Spark on Hadoop is extremely complex and clunky when compared to spark on kubernetes).
The issue is that most people nowadays have never worked with a bare metal Lamp server, so they grossly underestimate how large their company can grow before needing any distributed HA solution. I'd wager 90% of startups go bankrupt or are acquired before outgrowing a single Lamp server.
Plus at the very least, I'd be very careful about putting type systems into the same basket as Kubernetes. One is a basic language feature used offline and before deploying. The other is a highly complex interwoven web of tools that might take your systems offline if used incorrectly.
Without Kubernetes, you need Debian and it's Apache and MySQL packages. It's called a LAMP stack and for many production deployments, that's good enough. Because without all that "cloud magic", a $50 per month sever running a bare metal OS is beyond overpowered for most web apps, so you can skip all the scaling exercises. And with a redundant PSU and a redundant network port, 99.99% uptime is achievable. A feat so difficult, I'd like to mention, that Amazon Web Services or Heruko rarely manage to...
Complexity has high costs. Just because you don't see Kubernetes' complexity now, doesn't mean you won't pay for it through reduced performance, increased bug surface, increased downtime, or additional configuration nightmares.