My experience is the opposite. Most productions users can tolerate downtime.

It's usually the company leadership that can't tolerate it to have downtime.

And it's fact, they are hit an unplanned with downtime of one service or the other every month or so because of an outage. They are used to it.

So if you plan for it, explain it, and limit the scope and time of it, it usually goes very well unless you are a fortune 500, a hospital or something alike.

I think you really underestimate the amount of dependencies (documented and otherwise) that exist in even medium sized company.

I once caused a production outage in a retail company that caused all cash registers to stop working. The team that worked on that had pushed in a last minute change and didn't test if it handed going offline gracefully.

Right now I'm on call for an identity provider used by people in various timezones, including logistics workers in places that operate 24/7. Even when we do weekend upgrades, we still cause quite a bit of collateral damage. 10 minutes of time, multiply by the number of employees affected. It adds up fast.

I’d say a system not designed to support maintenance is not properly architected. You will need maintenance windows for an individual subsystem to perform OS upgrades, DB upgrades are no different and Postgres upgrades using pg_upgrade are relatively painless, provided you test them first in CI or against production snapshots, something the author’s company seemingly lacks.

"a system not designed to support maintenance is not properly architected"

Indeed, but who cares about the system design anymore? How many companies/ teams can claim honestly they even had a person with proper DBA competency, while features over features were added in sprints doing the minimum required to get the feature shipped out at the soonest possible (usually one DB schema change with feature and then one or more to add index due to performance regressions)? DBA competency is only sought when DB schema has fubar'd to an extent that frequent outages are norms or the version used is EOL'd by a few months at least. And by that time the people who "designed" the system are gone, not having documented ever why a given decision was made.

And that is how my friend...

DB upgrades are hard.

Indeed you can get away with a lot now but just paying (a lot) more money; it feels like design is no longer needed as it works. It is how I make some money; people come to me with; ‘we run some trivial online shop made by our team and with 100k uniques a month we pay $15k+ for rds, is that normal?’. So I go in and fix it. Usually it is bad or no design of the db schemas which was countered by picking heavy rds instances. Fun times considering I expect this to be the case in a very high % of all rds deployments (all that I have seen so far, so for me it’s 100%), not only the ones that asked me for help. When asked, the story is the usual ‘people are more expensive than hardware’ blah. It usually takes me around $1000 in fees to cut half the costs so that is blatantly false. Not to mention that the human costs are one off; if someone pays me 20k to bring 15k/mo to 1k/mo (which is typical), it is worth it. Unfortunately that’s not the break everything and go bankrupt way of working I guess! Still I notice that in current harder Financial Times, I do get more requests.

How happy are you when your global bank has downtime in middle of the day?

You wouldn't schedule downtime for the middle of the day. You schedule downtime during your period of least usage.

You design the complete system so it does not have dependencies on a single component, that way each subsystem can have proper maintenance schedules. It takes a lot more up-front work, along with defining what levels of service degradation are acceptable during a window, but that's the difference between a professionally run service and one run by amateurs. Look up Google's SRE methodology and their concept of error budget.

You mentioned hospitals yourself. There are also alarm management solutions for alarm receiving centres, and security and emergency services. Can't really have downtime there too. Of course there are less busy hours, but an alarm can come at any moment and needs to be handled right then.

Blue green deployment is, while it requires a professional, is not particularly complex or unusual.

Postgres is a tool. It can do many things, but the tool cannot run your entire business.

"Postres is a tool. It can do many things, but the tool cannot run your entire business."

this is something majority of the decision makers pretend to don't understand (and sometimes really don't).

zero-downtime-postgres-migration-done-right:

https://engineering.theblueground.com/blog/zero-downtime-pos...

Would you consider that a piece of cake? I have doubts. The page you linked to describes that if you follow the instructions on many other blogs you'll have data loss. Not a symptom of a piece of cake process

If you need zero downtime, you already in a field where NOTHING is a piece of cake. Not your network, not your computing, not your storage, and i haven't even talked about the human aspect of "zero downtime".

Alright. But the GP comment claimed upgrading PG was a piece of cake. You claim it is not a piece of cake. So it sounds like you agree that the claim that upgrading PG was a piece of cake was misleading.

> the GP comment claimed upgrading PG was a piece of cake. You claim it is not a piece of cake. So it sounds like you agree that the claim that upgrading PG was a piece of cake was misleading.

GP claimed upgrading was a piece of cake, not that zero downtime upgrades are a piece of cake. The two claims aren’t interchangeable. The simple upgrade path is always available, though it may have downtime consequences you personally are unwilling to accept. And the complex upgrade path is complex for reasons that have nothing to do with PostgreSQL - it’s just as complex to do a zero downtime upgrade in any data store, because in all cases it requires logical replication.

So if anything it feels like you’re the one being misleading by acting as though GP made a more specific claim than they actually did, and insisting that the hard case is hard because of PG instead of difficulty that’s inherent to the zero downtime requirement.

So if I tell you that upgrading pretty much all databases is a piece of cake but not include the criteria "unless you want to keep your data" you would say that is a fair statement?

If you claim that process X is trivial one has to make some assumptions, right? Otherwise I could claim that going to the moon is trivial but leave out "assuming you have a rocket, resources, people and anything else you may require".

Claiming that something is a piece of cake as a broad statement without any details is meaningless at best.

> So if I tell you that upgrading pretty much all databases is a piece of cake but not include the criteria "unless you want to keep your data" you would say that is a fair statement?

Incredibly bad-faith comparison, this.

Many, many datastore deployments can tolerate 10 minutes of downtime every 4 or 5 years when their PG install finally transitions out of support. Data loss isn’t even in the same universe of problem. It’s reasonable to talk about how easy it is to upgrade if you can tolerate a tiny bit of downtime every few years, since most people can. It’s utterly asinine to compare that to data deletion.

CockroachDB and others come with huge downsides in performance and features. I keep trying to see of these databases are a good fit for me but just the sheer time it takes to import my database makes me shudder. I've used the Aws cloud hosted free tier of CockroachDB but ran into transaction timeouts on very moderate loads. There is a reason these databases aren't seeing massive uptake despite their lofty promises.

Yeah CockroachDB suffers from the same problem as PG: the defaults kinda suck for certain workloads. CockroachDB has an additional problem that their documentation isn't as detailed as PG so it can be hard to know what to tune. TiDB and Spanner defaults are much better for a wider range of workloads.

Can you say more about workloads which suck for either of the two databases at their default settings?