Yep, drives me up the wall. We put a lot of work in making sure we're zero-downtime, pay for "Highly Available", and every couple of months I get the dreaded email. No explanation, nothing. Then we have to do the whole downtime dance.
If anyone from Cloud SQL is reading this, please, please stop doing this.
It’s insane that even with all of their HA and failover turned on they take the whole cluster down for as long as they like every few months!
To an engineer, things are things, because of how they architect and build them.
To an engineer who understands a customer, things are things and all the things people actually use them for. Big difference.
It also makes "Well, that customer is using it wrong" less of an exceptable engineering dodge.
I haven't worked inside AWS or GCP, but I've never seen product get everything they want, especially around maintenance/downtime. If "less downtime" is on the roadmap but engineering is constantly pushing back "that'll be really really hard and take a long time and they're just using it wrong anyway," I can't imagine it getting done as quickly as at a place where the engineering team was also focused on customer satisfaction.
It probably is hard and intensive. Engineering shouldn't lie and promise that it will be easier. Product has to take that engineering estimate and determine whether to work uptime or some sexy feature (and sexy features usually win because of perverse incentives).
Moreover, I have a hard time believing this for a couple reasons: first of all, I've scarcely met engineers who were opposed to improving product reliability, maintainability, etc. The portrait of Google engineers arguing that database services fundamentally shouldn't be HA (and customers are "using it wrong" for wanting HA DBs) is particularly incredulous. Secondly, I've never heard of an organization where engineering held political power over product decisions, but I have worked in several places where product dictated engineering solutions. Businesses trust product more readily than engineering because the things that engineering is always petitioning for are abstract and "costly" (deferring some immediate profit for reduced costs in the long run) while the things product wants are usually tangible and profitable.
If product has PM or financial incentives that reward without regard to technical debt...
If engineering has PM or financial incentives that reward without regard to user experience...
If you want people to cooperate, set up shared, team-based incentives!
Behavioral properties are end-to-end, like speed, security, customer success, uptime, etc. Everyone and everything on the hotpath for that has to be on board, as just one violator means nobody is achieving it. Operationally, part of achieving that means company norms, such as when collaborating, planning, executing, etc.
It's 100% fine not to provide any of these, and is even reasonable given it's hard to change one's DNA end-to-end... but better to be an explicit decision. Likewise, if starting a company... a lot easier to pick & ingrain these habits ahead of time.
Sure, you can have zone-redundant Azure SQL Databases, but not Azure App Service!
You can have zonal Azure App Service, but with a different network model than the default, so it's a breaking change for some apps.
It's as if nobody has actually sat down at Microsoft to build something similar to what their customers are building. It's all just tech demos, "get your blog hosted in 5 easy steps", and crap like that. Nobody at Microsoft has ever built anything of substance with the majority of their own platform.
It was the same story with Windows Presentation Foundation (WPF). It was hot garbage when it was first released, but Microsoft kept telling all their partners to prefer it over the legacy GDI+. People tried and failed to write GUIs in it. When after many years Microsoft finally tried to convert Visual Studio to WPF -- the first time they had used their own framework for one of their own apps -- magically the core issues were fixed and WPF become viable for real apps.
In general, Microsoft, Google and others are behaving really enterprise-y in that they are slow, you cannot reach anybody of importance to solve massive issues with their products and everything is actually quite expensive for what it does.
NO, RDS includes non-Aurora versions of many DBs, plus MySQL and Postgres-compatible Aurora servers, plus MySQL and Postgres compatible Aurora Serverless.
Which is fine. It can just be a little confusing as they drift and the caveats grow.
Current Aurora Postgres is compatible with pg 12.4; pg 10+ support has been around so long that several versions that support 10+ have already been EOL’d by Amazon. Even Serverless, which lags behind, is on 10.x.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
That said, this is not some poor rep's problem. This is a problem of Google culture and one that will be very difficult to fix. They simple don't care much about their customers and never needed to.
I just watched a load test hit that issue, the developers cranked up the traffic until the response times were measured in minutes. At that point Azure SQL started throwing errors related to too many concurrent queries.
Compared to just locking up or timing out, it's probably better.
See: https://docs.microsoft.com/en-us/azure/azure-sql/database/re...
And: https://docs.microsoft.com/en-us/azure/azure-sql/database/re...
E.g.: the 4-CPU pool has only 210 Max concurrent workers per pool, but if you think about it, that's over 50 queries running at once per CPU core, which is nuts!
Ps: if you’re working on a production load system, and has decent traffic, lean towards premium/business critical tiers for true HA
“Downtime as part of Scheduled Maintenance will not be counted towards any Downtime Period.” - https://cloud.google.com/sql/sla
I’ve never seen an SLO document for CloudSQL though so the SLOs may be slightly higher internally?
AWS tells you when your maintenance window for an RDS instance will be and you can delay/reschedule as needed.
I wouldn't consider a maintenance window as "downtime". That 99.95 applies to all the time outside the window.
When we first launched on GCP, there was no question that it was the way to go (frankly, because of BigQuery). Working with AWS, when we launched, was going to cost us significantly more up-front, before we had even brought on our first customer.
Fast forward 5 years... AWS has closed the gap in every way that matters. I still, frankly, trust Google's Security more than Amazon's, but I don't encourage folks to use GCP the way that I used to.
Just the opposite. In 2021, no questions asked, it's either AWS for general compute, or something more targeted if your business doesn't need it.
Until you get to the point where your bill is larger than a dozen engineering salaries, you won't get any respect from these people.
> I still, frankly, trust Google's Security more than Amazon's
If you have the time, could you expand on this? While I'm not directly involved in security at AWS, I'd be down to forward your thoughts to people who do.
My comments are mostly backed up by my experience at startups and are not colored by my experience at Google (too different a beast).
GCP is great for teams that are also using GSuite because you can set permissions at the level of a Google Group and have them propagate to individual members. You can, of course, also create groups in AWS but they don't have the same semantics of Google Groups and don't cover the wide range of use cases that Google Groups does.
The AWS scopes -> policies -> roles -> resources chain of abstractions is less natural conceptually than GCP's GSuite accounts + service accounts with attached scopes per project.
Also the fact that each managed service (GCE, GKE, Cloud Builder) has its own service account that you can attach scopes to is really nice. GCP service accounts just feel more discoverable than AWS IAM roles - I think it's because the number of AWS pre-built roles is so overwhelming.
Just some thoughts off the top of my head.
I’m sure they exist, but over the last dozen or so years I’ve worked with public cloud offerings across 5 or 6 industries and domains, I haven’t found a use case that can’t be easily implemented in the simpler GCP model.
AWS support is really nice. That I miss.
Top of mind things I found weak/weird: - Basically can’t do least privilege, only want a role to read messages from one pubsub queue? Nope not possible - IAM policies seem bolted on, legacy roles seem much better fit in the ecosystem, but they suck obvs. - different gcp resources have somewhere between very coarse grain to just acceptable iam operations. Might just be GCS that lets you do proper least privileges policies like you would do in AWS.
I was very surprised how bad it was, AWS IAM is some black magic shit that is deeply impressive and often taken for granted, even GCP can’t replicate.
But I agree, AWS IAM is some black magic and is very powerful.
The biggest problem I have with GCP is that something will say "you need the foo.bar.baz permission", and when I go to the IAM page to give that to myself... there is nothing in the search results for "foo", "bar", or "baz". Instead, I have to guess the "friendly name" for the permission.
https://cloud.google.com/iam/docs/understanding-custom-roles
Often the UI, and docs make it seem like everything is all over the place but AWS feels like lego with some pieces tucked away. That is where I think AWS can be improved upon with a better documentation UI and discoverability.
I do have to commend Google on Flutter + Firebase + Firebase Functions. I think if Amplify focused on serving Flutter users more it could pull me away from Google altogether.
Unfortunately, Google has done a fantastic job with making Flutter integrate with Firebase through Android Studio and there really is no product from AWS that matches its developer friendliness and low learning curve. This makes it very easy to switch.
I guess it is somewhat of a threat because the Firebase Cloud Functions also offer something of a counter to AWS Lambda as much as I love using it with API Gateway.
And telling them to "just learn it" isn't the customer focused mindset, it's the engineering one.
Custom roles created through Terraform helped a lot.
Yes its a bit of a pain having to add policies sometimes when you are first getting started but once it's up and running you can rest easy.
Learning it isn't that much more time consuming or difficult, its just a bit of effort that is all (we are talking a few hours at most).
The discussion is about GCP having much better functionality around it where projects and permissions are naturally integrated with gsuite organizations and users. This is objectively true. AWS has an archaic project system with a dozen different attempts at uniting it all but nothing comes close to GCP's smooth and easy manageability.
Integration with our org GSuite (this alone is a massive plus).
I trust virtually anyone's security over Google's. I've never had issues with AWS. I've consistently run into serious Google security failures. Google has airtight security for its own data, but not for its customers.
Examples range from Chromebook and Android security update policies (tons of expired machines on the public internet, in the case of Android, usually without people knowing), to pay-for-security on GSuite, to really difficult-to-audit Google Drive security (there's no convenient way to track and audit what was shared with whom or where data went), to just a ton of other things.
I've never seen Amazon be callous with my data. I've seen Google do things that even nineties "we don't need security" Microsoft wouldn't have imagined....
The only people I know who really trust Google security worked for or are close to people who worked at Google. There's a reality-distortion field based on how much Google invests in its own security that people fail to notice very basic failures, like millions of expired Android devices, or a lack of audit logs if someone physically accesses your machine to rifle through your gmail....
Nov 2019 - interviews
Dec 2019 - passed hiring committee
Dec to June 2020 - emailed recruiter every two weeks never heard back
June 2020 - cold email from recruiter “here is your offer letter to join Google! Let's talk team match.”
Needless to say I did not accept that job. This was for a software engineer role.
(edited to add line breaks)
Many weeks to generate offer and paper work and then 24 hrs to accept with a "take or leave it" ultimatum.
Just a very poor taste and I know some fine people who work at Google. Its unfortunate they don't seem to value warmth and humaneness when communicating externally
Noped out of that as quick as I could.
It crushed me because I found a team that I was unbelievably excited to work with.
Me and a friend of mine interviewed in Nov and passed HC in Dec 2019 as well (different recruiters between the two of us). Our experience was different from yours: Immediate timely followups from the recruiting coordinators, team-matched and hired in the same month. Hopefully our experience is more common...
Some European-based customer apparently had a requirement if we engaged with them, that our service be offered via an acceptable vendor such as GCP, for some reason AWS apparently wasn't, but it was such a nightmare to even prod about an architecture that would have feature-parity with AWS, it wasn't even worth it. Also as an fyi, I'm no AWS fanboy, I don't use it in any of my own projects to avoid vendor lock-in this company suffered from.
Since at least when we started spending about $100k/yr we've had a dedicated account rep we can contact at any time. They also get in touch to schedule a check-in every few months.
They've been genuinely helpful in several situations and have scheduled meetings with various teams around AWS (like, actual engineers) to get us answers to questions, support, and guidance. We've been put in touch with team leads and engineers working on beta features when we tried to use them and had issues to report.
Obviously this is all a sales tactic: if we have questions about X, putting us in touch with experts in X makes it more likely we'll successfully implement it and then pay them to use it. But it's the kind of sales we're getting value from, not just blindly pushing us to pay them more money.
We don't pay for any support package or anything.
Developer support $29/month, and business support is $100 (both go up if you spend more).
I paid for business support. You get
24x7 phone, email, and chat access to Cloud Support Engineers
Unlimited cases / unlimited contacts (IAM supported)
This is for $100/MONTH!! That is the deal of the century.
And they are ridiculously helpful.
I don't understand this - AWS must be losing money at least on support side, though they obviously get happy customers (myself included).
And even at $150 this would be great.
I had a client on gsuite with google 8 years ago - we COULD NOT get anyone to help with some weird admin state flow issue - it just was not possible to talk to a human being for ANY amount of money.
Working at a place with AWS enterprise support by contrast, for the second occasion, I would suggest that many of the places paying $15kpm don't cost that to support.
Google has treated me so badly so many times now on my personal account (as well as on business accounts, for that matter) in so many different ways that they've, conversely, lost MANY million dollars in business sales. It's hard to even count; a lot of people ask me for advise on decisions, and whenever someone even thinks about using Google Cloud in a business setting....
This is not a hole I see Google getting out of, except by eventually shutting down the Google cloud. Too many people have had too many bad experiences, and reputations take a long time to recover.
And the failures just keep on piling up.
Google is great for personal use, but I think they're diversifying in all the wrong directions. They're not structured for success there.
Like regional disks [1] or live migration of compute between hosts if problems are detected with the host.
When would you ever want to rely on this? Seems to me like you should have two hosts in the first place.
I'd guess they don't get many resource intensive support queries from the < $10k a month customers (and at that level you probably don't get the A team support)
The other poster indicated it is possible for stuff like this that you get bumped, even on the el cheapo plan, to someone actually on product team. While I'd hate to be on the product team having to answer these, it perhaps keeps folks aware of customer issues so they can at least improve docs for corner cases?
In general the AWS support has been great. In many cases, they've forwarded our requests to product teams who have even fixed bugs we've run into and contacted us directly.
Our other experience is with paid Azure support, which did little else than direct us to the (not related to the question) docs. They also had a really hard time understanding our technical questions about specific APIs. To their credit, they did eventually escalate to the PM of the service in question.
In general, the team responsible for the service really must be able to help out with support requests. In AWS this is definitely the case, in Azure as well but there's a bit of gatekeeping. Does developers and PMs in GCP participate in support?
Their model was fault-based, and a “bug” gets billed to the support group. So the game was always for MS to avoid assignment for non Sev-A cases, and our game was to find a product defect for anything.
Yeah... just like the other 5 dupes of the same suggestion with hundreds of upvotes that the product teams have dutifully ignored because it only matters to customers.
There are plenty of cloud agnostic platform out there that does VPS, load balancers. digital ocean, linode, etc.
If you want to be green there is also the Advania they use geothermal energy. And since it is in iceland better data protection policy than US companies.
I'm not affiliated with any one of them, it is just from year of cloud provider hopping.
Only thing I miss is firewalls for cloud (they have it for dedicated).
While everyone else is talking about game engines, design and programming techniques, Google's talks are mostly around their cloud offerings, and customer telemetry.
So why is Google this way? They already have a terrible reputation for their non-existent support. Never thought they'd treat actual companies this way though. It makes no sense...
Of course, now when they enter a business that needs direct customer interaction, it shouldn't be surprising that they aren't good at it. No supportive processes and probably no internal appreciation for what reps do. Thus the turnover. They will have to build out a sub-org that nurtures and develops this part of the business.
I personally find customer interactions problems to be epidemic with fast-scaling internet businesses. It does however create a space for others to come in and do it better. We can hope...
>> Hahah. They did that to me too. I doubt they even looked at it. It was a fun homework assignment though.
"Once I provided that to them I was met with radio silence."
>> Hahaha. Same here.
"which will inevitably devolve into the discussion of how can I use BigQuery"
>> This is a trap. Once you are on BQ there is no sane way off.
Experiences with support or lack of support concerning other Google product areas and divisions, especially those not designed for businesses really don't apply if you know how Google operates.
I don't use GCP in a personal capacity at this time and I work for a competitor (though am certainly not speaking on behalf of the competitor).
It is true there are individuals that shine, but across the board, Google sucks at support. It starts at the top with what kind of company they want to be, and goes from there.
They do not like humans.
You should have seen the extent to which the rest of the team and I tried to resolve customer issues.
Random App Engine app getting 500s calling an Open ID endpoint (the service wasn't strictly speaking managed by GCP). I absolutely could not reproduce this and had no other reports, but I knew it was happening for the customer. I examined everything that could be different... After 2 weeks I finally got it: The customer app was running in a different data center compared to where I was testing. Calls to the API endpoint were timing out internally (pretty aggressive timeout interval). It turns out the Open ID service wasn't running in the same data center region. That should not have been the case but occurred because the internal BigTable service had to be upgraded in a particular region and hence all App Engine apps in that region were moved to another one - a fairly seamless process. The Open ID service unfortunately wasn't provisioned in that region (it's not a dependency of App Engine). Once I identified the issue it was a matter of telling the on call SRE about it who then spun up Open ID in that region and thereby resolved the issues in a matter of 5 minutes.
Often a new support case meant discovering platform limitations - for example running into BigTable anti patterns because App Engine Datastore allocated IDs sequentially by default. That could cause poor read / write performance, in part due to automatic sharding behavior of BigTable. We later switched to snowflake IDs.
And then there were lots of Java folks used to Classpath scanning (default behavior of Spring Framework?) which for larger projects caused App Engine container startup to exceed the allowed time limit. That became a user education issue for which we then had to write an in depth article.
Good times :)
I absolutely hated dealing with billing inquiries though... so difficult to map someone's code to billable operations. Some SDK methods were of course utilizing multiple billable operations. Determining SLA violation refunds was much easier.
I'm not saying you didn't do great work, but I think the secret to success at AWS Support is normalizing this level of customer obsession.
We did whatever it took to resolve a support case in the quickest yet most satisfactory manner, even if it meant doing some crazy research, meanwhile we got of course overloaded with more support cases coming in.
I had 100% customer satisfaction from all surveys. That was definitely hard to achieve because happy people usually don't complete surveys. :)
These were the early days of many GCP services. I worked there when Compute Engine and Big Query GA'd. This was just before and possibly during the time the paid support tiers were launched (I don't recall). I left mid 2013. No idea what their support is like now.
And those technical account managers just bounced customer questions to us 3-5 support engineers (technical solutions engineer)
Anyways, your AWS comparison may apply to the GCP support culture now (I have no idea) but back then we were all extremely customer obsessed - by choice.
At the end of the day, I don't think the complaints you hear about GCP Support / Service has anything to do with the people working on the line. There's always better or not better examples of folks you encounter as a customer.
As a support consumer, I've learned that my feelings about the person on the other side are very much colored by the fact that I almost never call support when things are going well or I'm in a good place. I typically call support when all else fails and I'm probably ready to throw a product out the window. That's not their fault, and it shouldn't be their problem, either, to a certain extent. They're humans, doin' a job, just like me.
As a support provider, I've learned that the same is true for all of my customers. I need to know that this isn't who they are (usually) when they go home to their family, or out with their friends. They're under a lot of stress and, one way or another, we've let them down and have to deal with that.
I think the challenge with GCP support is all about the overall strategy as a business. My experiences with GCP support is that I have to work way too hard to get past the robots and the scripted responses in order to find someone who is willing to listen and engage with my question or problem. And then, to make matters worse, about 80% of my interactions with Google support involves them trying to introduce me to some consulting company or third party provider. That is pretty much never what I want. I don't want some third party billing integrator that is going to mess up my reporting with their "value added" service. I already pay for support, I don't want to hear a pitch about paying for more support when I feel like I'm not getting what I've already paid for.
I don't hold this against the support folks at Google.
I hold this against the leadership at Google.
This is their strategy to contain costs or to juice sales.
If it were working, I think their numbers would be very different in the market. So the proof is in the pudding.
Something isn't working at GCP.
I don't think it's the engineers or the teams. It's the leadership and the strategy.
A lot of support became outsourced and the original support team became more of a tier 3 backline support with the first 2 tiers being handled by vendors. This started for GCP just after I left in mid 2013.
The vendors responsible for support have contracts to meet certain customer satisfaction metrics and time to first response metrics (based on customer tier and ticket priority). The majority of inbound support cases would get auto assigned to the vendor team. Only the top customers would reach Google employees directly.
The vendor and their staff simply have no incentive to troubleshoot an unusual issue - they also cannot have access to the tools that may expedite such troubleshooting. It's all about pattern recognition to quickly resolve a ticket. As many vendors were overeager to escalate support cases to the Googlers it became necessary to impose more requirements on the vendor team to be able to escalate an issue - invalid escalations were being tracked and the vendor would be penalized for the total of these.
I built the internal tool via which GSuite and GCP outages or significant issues would be managed from a support perspective. This had the advantage that vendors didn't need to waste their time (poorly) parroting (with a delay) the status of an outage as related incidents could be (manually) grouped together and then all be updated at once by the current incident manager on the support team - freeing up lots of support folks to work on other support cases.
I definitely agree this vendor-based support model is a frustrating experience for everyone involved. I witnessed that myself after leaving Google. By the way, I'm very surprised GCP tries to upsell you on consulting services based on your support interactions.
Running a support organization for a very technical product with so many unknowns and variables at scale is hard. I wonder who does that well and how they accomplish it (from an operational perspective).
I have no idea how AWS manages support internally. I work at Azure - until last week as a Developer Advocate, now as a software engineer in Azure CTO incubations - I have no idea how our support organization is run or what their quality is. However, from what I hear Azure support is eager to help anyone and everyone. I remember being very confused I got a phone call to inquire whether I needed help within a day or two of signing up for an Azure free trial.
(no, I'm not and have never been a Google employee)
Back in the dotcom days companies would spend a fortune on Sun kit but I bet when averaged out over time a comparable company would be spending a LOT more on cloud billing.
I would like to learn more about this. I'd have thought costs should go down over time. Are we doing more or is the cost per unit (not sure what that means) is truly going up?
I think a number of factors add up over a 3-5 year timeline (obviously this will be more or less true for different organisations). There is the way the cost scales for a given instance in the cloud - in the old days, for example, doubling the memory or doubling the CPUs didn't double the cost of the kit, but it does for clouds VMs.
Another example is that the cloud bills you for everything, in the old days I could have a database server on a network and query it as much as I liked, the cost was fixed upfront for the lifetime of the hardware. Whereas it's very cheap to get started with a managed offering but e.g. BigQuery charges you for every query, Cloud Functions charge you per invocation, bandwidth is chargeable etc.
Speaking of hardware, in the old days you could look at your hardware and say, actually, it's fine, we don't need to upgrade/replace it this year after all, and it will just keep running. Whereas in the cloud the payment is continuous (perhaps offset by the fact that it's easier to "give back" excess capacity).
There will come a point at which the cost of DIY vs cloud will cross over, the question is whether you will reach that point, and if so, what you will do about it, since you may be well and truly locked in at that point.
You always can drive cost concessions from sales, especially for base workloads where you have time flexibility.
For a big company, cloud rarely saves money for many categories of expense. In a normal market, it is almost always faster time to market to rent, and always cheaper TCO to own.
This is a different market and one which is ultimately constrained by the availability of land. Notably, cloud prices do fall especially relative to the compute power. Specifically I remember when Fargate moved to firecracker and prices fell by like 40% or something similarly considerable.
Maybe managing your own internal cloud is indeed cheaper (especially if you don't account for support or maintenance!), but arguing that cloud prices don't decrease or making some housing analogy seems like poor reasoning.
It’s always better for you as a person to rent box truck to move. If you’re a company that needs a truck 3-5 times a month, there’s a probability that leasing may make more sense.
I’d say that businesses that suck at managing on-prem will not magically get competent in a public cloud.
It takes time to build a competency. If you have to figure out how to operate everything from the hardware and networking all the way up to application, then you might very well fail as a business. If you can pay Amazon to manage your networking, hardware, databases, managed services, etc while you work on your application-level competencies, you stand a much better chance of succeeding ("I don't know if I could build everything from the ground up, but I can probably fumble my way to a container image"). As you become very proficient, you can cut costs, and if you're super proficient you might get off a public cloud altogether or more likely you'll just use that as leverage to negotiate a lower cloud bill.
Interestingly, you don't hear about many businesses that are so competent that they move from the public cloud to their own private clouds. Some people will shout "Lock in!", but I don't buy that--I've migrated between public cloud providers before and it's work, but if you're competent enough to run your own private data centers then you're plenty competent enough to migrate to them.
GCE also isn't so much predatory as incompetent and apathetic. If a Google failure wipes out your startup, you're a statistic and it's okay.
The massive number of newer services are propriety, often buggy, and poorly documented.
I don't use any AWS services introduced past 2015 or so.
My company has gone through 4 reps in 3 years. Every time we get a new GCP rep they just want to talk to us about "expanding our use of GCP offerings". The only thing they want to talk about is starting to use BigQuery - not my business at all.
I signed up for a Google Cloud Security summit, and afterwards a sales rep reached out me. It was obvious from the start they had no idea I was a gsuite or GCP customer. They then directed me to a NEW account manager (#4) even though I had been working with a different one. I had worked with the prior account rep going over our architecture to make sure everything was kosher (sustained use discounts etc). I even made them a schematic of our architecture on GCP at their request. Once I provided that to them I was met with radio silence.
It's really insulting, and to me obvious they don't care about my company at all. We're looking at other options.
I wouldn't be surprised if somehow this post leads to me getting an email from another rep "Wanting to start over and doing things right", which will inevitably devolve into the discussion of how can I use BigQuery.