What you're asking for is an essay on comparative devops architectures, with a focus on k8s alternatives. I think what you'll find is a lot of ad hoc persistent systems that tend to drift over time in unpredictable ways, and take on the feel of a public lobby if you're being generous, a public restroom if you're not. So what you're asking is really a sample of these ad hoc approaches. What I think you'll find are a few inspired gems, and about 1000 bad implementations of K8s, Docker, Salt, Terraform and all the rest.

The case that always interested me is the service that can really fit on one machine. Machines are truly gigantic in terms of memory and disk. Heck, I worked at BrickLink which ran on two beefy machines that serviced 300k active users (SQL Server is actually quite good, it turns out). (A single beefy server is a really great place to start iterating on application design! PostgreSQL + Spring Boot (+ React) is a pretty sweet stack, for example. By keeping it simple, there are so many tools you just don't need anymore, because their purpose is to recombine things you never chose to split. I can't imagine why you'd need more than 16 cores, 128G of RAM and 10T of storage to prove out an ordinary SaaS. That is a ferocious amount of resources.)

>I can't imagine why you'd need more than 16 cores, 128G of RAM and 10T of storage to prove out an ordinary SaaS. That is a ferocious amount of resources.)

In my case because I was working for a checkout solution for a Cash & Carry chain, with 1000 stores in 25 countries, each with tens of tills, self service points, queue busting tills who also had to serve the web shop. And that solution had to work also when there was no Internet access.

Your project is significantly more complex than an 'ordinary SaaS'. My own canonical example of a SaaS would be something like WuFoo, or Squarespace. These are characterized as pure webapp component producing technologies, from soup to nuts, with through-the-web tooling for everything from specializing your components, to managing your payment details. The system is "closed" in the sense that it only produces and consumes web components for all operations. There is no real-world device that is critical to operation - so you are free to run services like these on any device, or collection of devices, as long as its users can find 'a useful stateful service' at 'name'. For your project, this is only the first foundational requirement.

Availability?

Devops is not my area of expertise to put it mildly, but "a single big-ass server" scenarios are often pretty well covered by "a second big-ass server configured as a mirror/hot-spare", right?

Depends on how many nines of uptime you need, of course, and other such things.

(Obviously, there are reams of cases where this isn't sufficient)

In 2009 I had a pair of (what I thought at the time) very cleverly architected VM servers with complete redundancy between them. Either one could be pulled and there'd maybe be a few seconds of data loss at the worst. One day lightning struck the building and even managed to "jump" the grounded pair of UPS's and fried all hard disks in both nodes.

Thankfully I had both onsite and offsite backups; but it took a couple of weeks to order the new equipment, install it and set everything up again.

Now I "devops" all the things and always have the ability to spin up all the services and restore from backups within minutes. This is the new standard in my opinion.

Two questions:

1. How would power conditioners and lightning arresters handle these sorts of situations?

2. Was this using VMWare fault tolerant VMs, just out of curiosity? (It continually live-migrates one VM to a hot spare over a 10G link so either can disappear and the VM continues to run.) Or was this a bespoke application architecture implementing effectively the same thing?

I think it's always better to have the mirror be located in a separate datacenter and then connect both via a SDN.

Why not both?

Yeah, there definitely needs to be an offsite spare for sure.

Why isn't "we're going to risk downtime to speed up our BTD loop for cheap" a good answer?

I've made that very decision.

Partly forced due to internal resource constraints. However, swapping getting a working system out and tested with real users instead of waiting an undefined time to get high availability didn't lose me any sleep.

It's also the case that some systems can withstand a degree of downtime that others can't, or it's not worth paying the cost for the perceived benefit gained.

>So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table

Decades of work typically. That or just standard parallel deployments, most things Docker is good at would take the average developer many unnecessary hours to reproduce.

Currently there is a maltitude of solutions available on the market that make containers and k8s just a complex and hard to maintain choice.

It all boils down to “what bullshit was my management sold on”.

Promises of “easy to use, easy to replace, cloud-agnostic, doesnt require domain knowledge” etc etc etc.

Its all bullshit. At some point/scale you need experts in the field that cash in a month more than average dev per year.

> Attacking the premise isn't particularly hard.

The linked Drew Devault seems to have a glib response for quite a few things. His devs shouldn't build software for users take is just as infuriating. You just want to say "Yeah, I'd rather not, but unfortunately it kinda goes along with writing software. Send 1000 happy users to me to build my software on every platform, because I'd be glad to let them. Otherwise, what is your alternative?"

> So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table

It's interesting that you're not interested in what they get by not using containers...

What? That's the entire point.