> "highly available" Cloud SQL goes down every couple months for maintenance

Yep, drives me up the wall. We put a lot of work in making sure we're zero-downtime, pay for "Highly Available", and every couple of months I get the dreaded email. No explanation, nothing. Then we have to do the whole downtime dance.

If anyone from Cloud SQL is reading this, please, please stop doing this.

We’re currently migrating to Spanner for a variety of reasons - but the mandatory downtime on their Postgres CloudSQL offering will be the part I miss the least.

It’s insane that even with all of their HA and failover turned on they take the whole cluster down for as long as they like every few months!

One thing that customer obsession at an entire-organization-depth level does is encourage broad customer use awareness.

To an engineer, things are things, because of how they architect and build them.

To an engineer who understands a customer, things are things and all the things people actually use them for. Big difference.

It also makes "Well, that customer is using it wrong" less of an exceptable engineering dodge.

Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency?

> Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency?

I haven't worked inside AWS or GCP, but I've never seen product get everything they want, especially around maintenance/downtime. If "less downtime" is on the roadmap but engineering is constantly pushing back "that'll be really really hard and take a long time and they're just using it wrong anyway," I can't imagine it getting done as quickly as at a place where the engineering team was also focused on customer satisfaction.

> that'll be really really hard and take a long time

It probably is hard and intensive. Engineering shouldn't lie and promise that it will be easier. Product has to take that engineering estimate and determine whether to work uptime or some sexy feature (and sexy features usually win because of perverse incentives).

Moreover, I have a hard time believing this for a couple reasons: first of all, I've scarcely met engineers who were opposed to improving product reliability, maintainability, etc. The portrait of Google engineers arguing that database services fundamentally shouldn't be HA (and customers are "using it wrong" for wanting HA DBs) is particularly incredulous. Secondly, I've never heard of an organization where engineering held political power over product decisions, but I have worked in several places where product dictated engineering solutions. Businesses trust product more readily than engineering because the things that engineering is always petitioning for are abstract and "costly" (deferring some immediate profit for reduced costs in the long run) while the things product wants are usually tangible and profitable.

Sounds familiar. Product and sales areas of the business look at immediate revenues - and hopefully profit - but are always more keen to do that at the cost of increasing technical debt within the engineering side of the business. Unless sales see a real impact to technical debt, they will always choose the short-term approach.

Agreed, and I don't think that's even necessarily the worst thing. It just means that engineering and product/sales have to have a conversation, mutual trust, and a shared vision that extends beyond the next quarter. These are hard things to cultivate, however.

Most of the places I've seen this go bad were because one side was incentivized to not care about the other.

If product has PM or financial incentives that reward without regard to technical debt...

If engineering has PM or financial incentives that reward without regard to user experience...

If you want people to cooperate, set up shared, team-based incentives!

I agree. In whichever case, blaming individual engineers seems weird. Rarely are individual engineers to blame, and even when they are the organization should be robust enough to route around the occasional deficient engineer.

It's a full-company culture problem where ~everyone has personal responsibility.

Behavioral properties are end-to-end, like speed, security, customer success, uptime, etc. Everyone and everything on the hotpath for that has to be on board, as just one violator means nobody is achieving it. Operationally, part of achieving that means company norms, such as when collaborating, planning, executing, etc.

It's 100% fine not to provide any of these, and is even reasonable given it's hard to change one's DNA end-to-end... but better to be an explicit decision. Likewise, if starting a company... a lot easier to pick & ingrain these habits ahead of time.

I tried explaining this to some Azure product teams, and they gave me a blank stare in return.

Sure, you can have zone-redundant Azure SQL Databases, but not Azure App Service!

You can have zonal Azure App Service, but with a different network model than the default, so it's a breaking change for some apps.

It's as if nobody has actually sat down at Microsoft to build something similar to what their customers are building. It's all just tech demos, "get your blog hosted in 5 easy steps", and crap like that. Nobody at Microsoft has ever built anything of substance with the majority of their own platform.

It was the same story with Windows Presentation Foundation (WPF). It was hot garbage when it was first released, but Microsoft kept telling all their partners to prefer it over the legacy GDI+. People tried and failed to write GUIs in it. When after many years Microsoft finally tried to convert Visual Studio to WPF -- the first time they had used their own framework for one of their own apps -- magically the core issues were fixed and WPF become viable for real apps.

My experience with MSFT is that the account reps are clueless by design. Even the sales assistants with fancy technical titles are short of clues. Easier for them to oversell when they don’t know what’s really going on underneath. We have to go 3 or 4 deep into their product org to find someone who will fess up to product issues that we are already aware of.

Also they now require you to become Microsoft Partner, if you want to use the oauth2 login not only for personal Microsoft accounts but also other Microsoft accounts such as those provided by a business running Office 365. They changed it ~ 3 months ago and are now not able to verify people in time. Even if they just want to check your name, email and maybe a profile picture and nothing else. The process is much less obvious than for the same thing with Google or Facebook. Even the ZIP-Code has to include spaces as if it was written on a letter, otherwise you will not be able to send the form, aaaand there is no hint about this requirement.

In general, Microsoft, Google and others are behaving really enterprise-y in that they are slow, you cannot reach anybody of importance to solve massive issues with their products and everything is actually quite expensive for what it does.

I had no idea this was a thing, is there any justification at all for this behavior? Sounds like a massive dealbreaker to me.

I use Aurora just for this reason. RDS is just a great product. Doesn't even need the cloud sql proxy equivalent since you can just install the authenticator into the database.

Aren't RDS and Aurora two different things?

> Aren't RDS and Aurora two different things?

NO, RDS includes non-Aurora versions of many DBs, plus MySQL and Postgres-compatible Aurora servers, plus MySQL and Postgres compatible Aurora Serverless.

Aurora is one of the DB things you can launch in RDS.

Thanks for clarifying. I'd associated RDS with the non-Aurora offerings of Mysql and Postgres. Judging by upstream version compatibility it appears Aurora is more heavily forked than their non-Aurora siblings.

I don't think Aurora MySQL/Postgres are just forks, I think they are a completely custom datastore behind a MySQL or Postgres-compatible interface (which probably uses a lot of non-engine code from the open-source base database.)

Regardless it looks like anyone choosing Aurora should not hold their breath for Mysql 8 or Postgres 10+ compatibility. Seems like only one major version bump has happened since they launched the first one (Mysql 5.6-to-5.7).

Which is fine. It can just be a little confusing as they drift and the caveats grow.

> Regardless it looks like anyone choosing Aurora should not hold their breath for Mysql 8 or Postgres 10+ compatibility.

Current Aurora Postgres is compatible with pg 12.4; pg 10+ support has been around so long that several versions that support 10+ have already been EOL’d by Amazon. Even Serverless, which lags behind, is on 10.x.

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

Wonderful! Not sure how I missed it the first time. Thanks for correcting me.

I use it in Postgres 12 mode.

WAIT. Be careful. That is a super expensive product with a high likelyhood of lockin. It doesn't support all SQL features. Also! I run hundreds of GCP databases and never ran into: "but the mandatory downtime on their Postgres CloudSQL", maybe it is only Postgres?

Same here. I have sent my GCP rep 5-10 follow up emails by now and still no response. It really feels like I have no one to talk to over there and I'm spending thousands per month.

Hit me, more than happy to make fun of the salient GCP rep for you in a way they really won't like :)

Curious what your approach would be?

That said, this is not some poor rep's problem. This is a problem of Google culture and one that will be very difficult to fix. They simple don't care much about their customers and never needed to.

I’ve seen both! Plenty of culture issues, or tools, and sometimes someone’s just checked out...

Bit of a surprising comment coming from the CTO of a large Google partner?

https://sada.com/about/leadership-team/miles-ward/

I think you’ll find me surprising :) It’s surfacing and chasing down root cause of these misses that make things better for all customers, and that’s my goal :)

What do you think partners are for if not this?

I helped a major user of GCP migrate off the platform to AWS for this exact reason. Totally insane that they still do this when AWS has had a rock solid offering in the form of RDS for like 10 years now.

Does RDS not do this to their customers? This alone would make me seriously consider taking my business away from GCP.

Absolutely not. I'm sure it's happened but I have never personally seen an RDS instance go down, in over 10 years consulting and building in AWS. Google Cloud SQL was going down for multiple minutes every month for one of my clients, and their support just said there was nothing they could do about it. That cost Google cloud at least a million dollars in revenue over the next few years as this was a fast growing startup.

I’ve been a long time Azure user, their SQL Azure product was very glitchy and had constant outages (“just add retries!”), it’s gotten better in the last year or so, but it was totally unacceptable.

AFAIK that's a "feature", not a bug. They throttle concurrent requests and return a different error code compared to the on-premises SQL Server.

I just watched a load test hit that issue, the developers cranked up the traffic until the response times were measured in minutes. At that point Azure SQL started throwing errors related to too many concurrent queries.

Compared to just locking up or timing out, it's probably better.

See: https://docs.microsoft.com/en-us/azure/azure-sql/database/re...

And: https://docs.microsoft.com/en-us/azure/azure-sql/database/re...

E.g.: the 4-CPU pool has only 210 Max concurrent workers per pool, but if you think about it, that's over 50 queries running at once per CPU core, which is nuts!

I’ve experienced that too, but these errors were literally “server is currently unavailable” and downtime.

Ps: if you’re working on a production load system, and has decent traffic, lean towards premium/business critical tiers for true HA

Do y'all just like, ignore SLO's? Cloud SQL has a 99.95% SLO, going down for multiple minutes every month is within that. No smoke and mirrors here, there are ways to mitigate it but it's not Google messing around with expectations. HA doesn't mean 100% uptime.

It’s worth noting that the dreaded maintenance everyone is talking about here is not part of the SLA:

“Downtime as part of Scheduled Maintenance will not be counted towards any Downtime Period.” - https://cloud.google.com/sql/sla

I’ve never seen an SLO document for CloudSQL though so the SLOs may be slightly higher internally?

Does that include or exclude "maintenance windows".

AWS tells you when your maintenance window for an RDS instance will be and you can delay/reschedule as needed.

I wouldn't consider a maintenance window as "downtime". That 99.95 applies to all the time outside the window.

Yet AWS manages to give near 100% uptime with RDS at a similar price point. CloudSQL downtime in the cases I've seen was caused by them rebooting both master instances at the same time. This is amateurish and totally unnecessary, the whole point of having multiple masters is to the ability to do maintenance reboots in a staggered schedule. This should be a trivial problem for Google to solve and would result in much better reliability for their users, yet it's been years with no change.

I had some instances that needed to be rebooted because they were running a vulnerable MySQL version. AWS gave six months warning (before forced reboot) and ended up extending it to 12.