News for hackers. Stuff that matters.
Re:“GCP automatically LOWERED our quota, causing an incident“ by jtchang on Saturday June 10, 2023 @02:34PM (#36273457)
Why don't I hear about these incidents with AWS? Is it just less common or different business takes on quotas and rate limits?
Re:“GCP automatically LOWERED our quota, causing an incident“ by rhtgrg on Saturday June 10, 2023 @02:45PM (#36273598)
> Is it just less common or different business takes on quotas and rate limits?

Neither, you're probably not seeking out such stories (there are plenty, even on HN). Google tends to get more negative attention on this particular forum, which might play a role, but the larger component is probably just chance.

Re:“GCP automatically LOWERED our quota, causing an incident“ by CSMastermind on Saturday June 10, 2023 @02:59PM (#36273765)
I disagree that it's simple selection bias.

AWS is _extremely_ customer friendly and if this happened would likely be offering dedicated support to make it right, credits for the business loss, etc.

Google's customer service is the worst I've experienced in the industry (like even speaking to a person is hard). While AWS is some of the best I've received.

Re:“GCP automatically LOWERED our quota, causing an incident“ by neilv on Saturday June 10, 2023 @03:59PM (#36274465)
I'm sure that GCP could turn around their HN customer service reputation, if they wanted to prioritize that.

Though, when it might not be baked into the culture as much as at AWS, I don't know offhand how to reconcile a customer happiness turnaround with the rush to avoid being an also-ran in the AI deployment frenzy.

(Are you going to assign conflicting KPIs in a company that has cultivated a career-driven culture, and hope that individuals strike optimal balances for the company? Or partition the goal assignments to separate teams, when the optimal outcome requires holistic thinking and behavior across teams?)

Re:“GCP automatically LOWERED our quota, causing an incident“ by etse on Saturday June 10, 2023 @05:08PM (#36275116)
Wasn’t this supposed to be Kurian’s prerogative—sell enterprise services? Google product seem fine to me, but I agree that they have no customer care DNA. I was expecting someone from the outside to bring it in.
Re:“GCP automatically LOWERED our quota, causing an incident“ by pram on Saturday June 10, 2023 @05:49PM (#36275518)
I worked in Kurian’s org at Oracle and I was shocked when Google hired him. OCI was a total disaster at the start. Like embarrassingly bad, they threw most of the original platform in the trash.

Literally failing upward.

Re:“GCP automatically LOWERED our quota, causing an incident“ by rhtgrg on Saturday June 10, 2023 @04:00PM (#36274476)
> AWS is _extremely_ customer friendly and if this happened would likely be offering dedicated support to make it right, credits for the business loss, etc.

It doesn't take much legwork to find counterexamples of that claim, even on HN (see below, I spent 2 minutes searching to find those).

> Google's customer service is the worst I've experienced in the industry (like even speaking to a person is hard). While AWS is some of the best I've received.

This is a bit too hyperbolic for me, but on balance I agree that Amazon has better customer experience than Google. That doesn't really answer what GP is asking, especially given all the posts made about AWS on this very forum that only sometimes get attention.

https://news.ycombinator.com/item?id=2478129

https://news.ycombinator.com/item?id=25224220

https://news.ycombinator.com/item?id=35375558

https://news.ycombinator.com/item?id=16283547

Re:“GCP automatically LOWERED our quota, causing an incident“ by marcinzm on Saturday June 10, 2023 @04:16PM (#36274627)
>That doesn't really answer what GP is asking, especially given all the posts made about AWS on this very forum that only sometimes get attention.

If these are your examples of angry AWS posts then I think it just proves that AWS has amazing in customer service.

>https://news.ycombinator.com/item?id=2478129

That's 12 years old but a legitimate complaint about AWS customer service.

>https://news.ycombinator.com/item?id=25224220

Sounds like AWS had an outage, not sure how this is about customer service?

>https://news.ycombinator.com/item?id=35375558

That's a list of AWS customer's having security incidents and, again, nothing about AWS customer service.

>https://news.ycombinator.com/item?id=16283547

Not sure what this is about since the link is dead except due to being 5 years old but, based on the upvotes, it's someone failing to get people to have angry opinions about AWS? Not sure how this is about AWS customer service given the context.

Re:“GCP automatically LOWERED our quota, causing an incident“ by rhtgrg on Saturday June 10, 2023 @04:28PM (#36274755)
Given that this isn't handpicked and just top results from a 2-minute search:

> That's 12 years old but a legitimate complaint about AWS customer service.

Should have at least prompted you to do your own search. The fact that it didn't is the end of this discussion.

Re:“GCP automatically LOWERED our quota, causing an incident“ by marcinzm on Saturday June 10, 2023 @04:33PM (#36274803)
So you made a claim that there's "there are plenty, even on HN" and then utterly failed to provide any proof even after trying. Then when called out you blamed the other person for not doing so on your behalf. I'm going to stick with my response.
Re:“GCP automatically LOWERED our quota, causing an incident“ by kenhwang on Saturday June 10, 2023 @05:04PM (#36275089)
I've seen my org send a silly amount of silly questions that were obviously our fault to AWS support and they always took it in stride. We very rarely needed to follow up with our client executive (but they always offer to jump in if necessary). When we did have serious issues, we had no problem scheduling time with the executive managing the product and the engineers that wrote the code.

Meanwhile we can't even get Google to answer an email about serious incidents that were very obviously their fault, much less assign a dedicated human point person (hell with AWS, we even had several backup contacts assigned to cover for vacation time). We were important enough to be featured on their client success frontpage, but that didn't make a difference in support quality we received.

I can't imagine why anyone would risk their business with GCE. Especially since it tends to cost more than AWS these days.

Re:“GCP automatically LOWERED our quota, causing an incident“ by whoknowswhat11 on Sunday June 11, 2023 @09:29AM (#36281192)
Yes, AWS being patient with both customer fault (common) issues and even out of scope for support tier level is crazy - it’s got to be a purposeful cost center expenditure. I like some things about google w project model but we spend on AWS (not too much).
Re:“GCP automatically LOWERED our quota, causing an incident“ by slowmovintarget on Saturday June 10, 2023 @05:07PM (#36275111)
Amazon worked very hard to build a culture of customer service. Google worked very hard to create tech and create a developer-first culture.

You're seeing culture in operation.

Re:“GCP automatically LOWERED our quota, causing an incident“ by bushbaba on Sunday June 11, 2023 @01:12AM (#36278331)
AWS would also have a GM be involved for such an incident, and if this happened more than a few times the GM would risk being fired.

Amazon really is customer obsessed. If a department fails its customers that departments leadership will be replaced

Re:“GCP automatically LOWERED our quota, causing an incident“ by oldtownroad on Saturday June 10, 2023 @02:47PM (#36273607)
Amazon hate their employees. Google hate their customers.
Re:“GCP automatically LOWERED our quota, causing an incident“ by neilv on Saturday June 10, 2023 @04:09PM (#36274564)
Best of both worlds would be a place that takes great care of both customers and employees.
Re:“GCP automatically LOWERED our quota, causing an incident“ by yjftsjthsd-h on Saturday June 10, 2023 @05:12PM (#36275149)
Well, yeah, obviously if you can have the best of all worlds then by all means do that, but in practice something's going to give; if you did somehow prioritize customer and employee well-being, it would almost certainly manifest in much higher prices, not actually sustaining quality, or both. (Now, charging more for a better outcome does seem like a good idea, but in practice >90% of customers will go for the cheaper option.)
Re:“GCP automatically LOWERED our quota, causing an incident“ by robocat on Saturday June 10, 2023 @05:47PM (#36275503)
Most people seem to equate employee well-being with more money.

Plenty of people care more about other job qualities than how much it pays, and a well run business can create a great environment without paying more than average. Look for companies where turnover is almost zero (assuming company is not growing) and you often find happy employees.

Places with worse employee conditions have to pay more, everything else held equal.

Anecdotally, I have certainly stayed in jobs with poorer pay because I liked my colleagues and the working conditions; but alternatively I have stayed in another job mostly because the pay was great.

Re:“GCP automatically LOWERED our quota, causing an incident“ by pclmulqdq on Saturday June 10, 2023 @07:08PM (#36276230)
How about someone that hates both? That would explain Microsoft's rise in cloud.
Re:“GCP automatically LOWERED our quota, causing an incident“ by koolba on Saturday June 10, 2023 @10:19PM (#36277316)
No man can serve two masters: for either he. will hate the one, and love the other; or else. he will hold to the one, and despise the other.
Re:“GCP automatically LOWERED our quota, causing an incident“ by iot_devs on Saturday June 10, 2023 @03:01PM (#36273797)
I personally reply to customers inquiries on a very famous AWS product where the client was asking why the execution was interrupted after the timeout they set themselves expired and why it was a timeout error.

I am an engineer working on such products and it was a routine on-call shift where I got this kind of question.

Admittedly we should have this cover by our support engineers, that are very very good, but this one slip through and I took the time to answer such query.

I am not a fanboy or anything, I could not be further from a fanboy or a blind fan.

But after working on AWS, I do suggest it as very sensible choice. Again, not because it is my employer but because they really take care of operations and customers.

Screw ups can happen but they are very rare in my experience.

Re:“GCP automatically LOWERED our quota, causing an incident“ by gurchik on Sunday June 11, 2023 @11:08AM (#36282082)
My only problem with AWS support is that they start every ticket assuming it is a basic, RTFM-type of inquiry and it can take a bit of effort to guide them toward more helpful responses. Once you get there they eventually get to your answer.

Here is a ticket I opened recently (paraphrased):

> Me: I'm seeing an S3 charge in my billing and usage report called "AMZN-Out-Bytes" what does this mean? I already read the list of billing codes in this page (link) and it's not there.

> Support: Actually the majority of your report is "DataTransfer-Out-Bytes" charges which is caused by egress traffic to the internet. For more information see (link I already gave them).

> Me: Thanks but I don't need assistance with that billing code, please explain "AMZN-Out-Bytes." And can you explain how it's different from "AWS-Out-Bytes" as explained in (link)?

> Support: As explained in the page you linked, "AWS-Out-Bytes" refers to data transferred between different regions in the AWS network.

> Me: Please explain "AMZN-Out-Bytes."

> Support: Please close this ticket and reopen it with the S3 team. (who ended up answering my question)

In comparison, Google support could only be described as hostile, seemingly looking for any reason to close the ticket as my fault.

Re:“GCP automatically LOWERED our quota, causing an incident“ by JoshTriplett on Saturday June 10, 2023 @03:22PM (#36274071)
I've seen a lot of incidents reported about AWS involving quotas, in the other direction: "I can't sign up new customers because I'm at a quota limit on resources", sometimes with "and AWS is taking a long time reviewing a quota increase", with the occasional "and AWS denied a quota increase".
Re:“GCP automatically LOWERED our quota, causing an incident“ by marcinzm on Saturday June 10, 2023 @03:39PM (#36274254)
Google wouldn't increase my quota unless I talked to a sales person. I wanted 4 GPUs. 4. That took at least a week to achieve.
Re:“GCP automatically LOWERED our quota, causing an incident“ by coredog64 on Saturday June 10, 2023 @04:58PM (#36275040)
AWS quotas come in hard and soft varieties. A hard quota cannot be changed as it’s factored into service operations (e.g. APIGW’s 29 second integration timeout). A soft quota may or may not require review before being granted. However, if it is a soft quota, the best thing to do is escalate to your TAM after opening with the business impact of the request. The TAM is paid out of your enterprise support contract, so the least they can do is advocate for you.
Re:“GCP automatically LOWERED our quota, causing an incident“ by cyclotron3k on Saturday June 10, 2023 @06:04PM (#36275643)
My problem with the quote system is that it's largely opaque until you run into it. Yes, you can pore through the documentation and find the relevant quotas, but did you find them all? Spoiler alert: no, you didn't.

I know they are trying to improve visibility on quota limits, and they have a tool now, but in my experience it was half baked and only knew about a handful of the limits we were running into.

Re:“GCP automatically LOWERED our quota, causing an incident“ by joecool1029 on Saturday June 10, 2023 @06:24PM (#36275821)
Some of the AWS services have clear quota limits and it works as expected, you give me 10k widget credits and if I use 10k widget credits it stops working and I need to open a ticket. Example of this would be SES.

Where it ends up being a bit of a nightmare is stuff like SMS on SNS where they would specify a quota but we could still hit it before their system reported it. We never did figure out if they were looking at a rolling monthly average and bursty campaigns could cause it to creep above their projection. This was always a manual review for this product and the only way we could avoid it was getting way higher quotas than we needed approved. Ultimately we ended up moving SMS over to Nexmo (now Vonage) for bulk SMS so we didn't have to have potential outages when the mystery quota of the now was reached.

Re:“GCP automatically LOWERED our quota, causing an incident“ by bushbaba on Sunday June 11, 2023 @01:16AM (#36278352)
Many quota systems are measured in qps cannot exceed the avg allotment. Hence yes, short lived large bursts deviating might not be allowed. Docs on the quota usually clarify this.
Re:“GCP automatically LOWERED our quota, causing an incident“ by throwaway2990 on Saturday June 10, 2023 @10:01PM (#36277218)
They have a quota page in aws console which displays all quotas. Defaults are in all the documentation. And quota information can be queries from the cli and will tell you if it’s adjustable or not.

They have a support page for quota changes and are usually actioned in around 15-30m on average.

What more do you want?

Re:“GCP automatically LOWERED our quota, causing an incident“ by adrr on Saturday June 10, 2023 @03:47PM (#36274334)
Because Amazon will jump through hoops to make the customer happy. If I file ticket to get a quota raised it usually happens the same day and I have never had them push back on me.
Re:“GCP automatically LOWERED our quota, causing an incident“ by theptip on Saturday June 10, 2023 @04:04PM (#36274507)
Ran my startup on GCP. No regrets. Can’t recall a time I waited more than a few hours for a quota bump.

Only time I ever got pushback was trying to get them to make an unreasonable quota increase to work around a bug in early GKE that left stale network backend lying around, and when I explained they approved it.

I’ve found GCP support to be reasonable, but you have to pay for the enterprise tier. Their pricing model used to be insane ($250/seat??) but they fixed that a few years ago. I suspect a lot (not all!) of the complaints around here are for hobbyist / free tier, which sure, is garbage. If you spend $K/yr on support it’s fine though.

Re:“GCP automatically LOWERED our quota, causing an incident“ by aledalgrande on Saturday June 10, 2023 @09:40PM (#36277121)
Same. As a complete nobody account I got access to (more) large GPU instances, even before the GPT era, without even getting asked why I needed them.
Re:“GCP automatically LOWERED our quota, causing an incident“ by balls187 on Saturday June 10, 2023 @05:04PM (#36275086)
AWS is great.

Gotta check out FBA/Amazin Marketplace to find all the nasty on Amazon.

Re:“GCP automatically LOWERED our quota, causing an incident“ by elankart on Saturday June 10, 2023 @05:14PM (#36275176)
In what world is AWS great? A world where they don’t have a clean way to group resources and clean them up? Their own twisted identity products in AWS that doesn’t integrate outside their cloud or their other products. A world where they leave lingering resources when you cleanup something.

AWS has to much hype riding behind it.

It’s not enterprise class and looks very incohesive.

Re:“GCP automatically LOWERED our quota, causing an incident“ by ripper1138 on Saturday June 10, 2023 @08:27PM (#36276780)
Is this comment from 2015? AWS just had a $20B quarter.
Re:“GCP automatically LOWERED our quota, causing an incident“ by balls187 on Saturday June 10, 2023 @11:22PM (#36277678)
> In what world is AWS great?

Mine. And Gartner’s.

I use GCP and AWS, and AWS outclasses GCP handily.

Re:“GCP automatically LOWERED our quota, causing an incident“ by whoknowswhat11 on Sunday June 11, 2023 @09:35AM (#36281227)
The point about lack of projects on AWS I agree with if that’s what you are talking a about. That said AWS is absolutely a cloud leader. We use azure and AWS now and there continue to be reasons AWS does well despite cost and quirks