News for hackers. Stuff that matters.
Re:Tell HN: Google Cloud suspended our production projects a... by baobabKoodaa on Monday August 22, 2022 @04:46AM (#32548901)
Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.
Re:Tell HN: Google Cloud suspended our production projects a... by doix on Monday August 22, 2022 @05:06AM (#32549055)
It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car.

Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.

Re:Tell HN: Google Cloud suspended our production projects a... by bastawhiz on Monday August 22, 2022 @06:53AM (#32549739)
How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway? Having a relationship with them doesn't matter if they still can't be trusted to do business. The type of payment method you use is not a factor that plays into whether you can trust your vendor.
Re:Tell HN: Google Cloud suspended our production projects a... by mcv on Monday August 22, 2022 @08:19AM (#32550315)
I don't think you can. My takeaway from this discussion is definitely that GCP is only suitable for toy projects, and not for serious production stuff that requires continuity.
Re:Tell HN: Google Cloud suspended our production projects a... by mhoad on Monday August 22, 2022 @08:58AM (#32550696)
This is a fundamentally unserious point unless you want to add more qualifiers.

I’m pointing to this as evidence https://cloud.google.com/customers

I feel like a lot of the problems that people run into with GCP are avoided with two main components.

Number one you should look seriously at the “enterprise” designation for different products and services if you are looking for long term stability and guarantees about the future (details: https://cloud.google.com/apis/docs/resources/enterprise-apis) and secondly you need to buy actual customer support as an add on if you want to avoid the “I had to hit the front page of HN to actually speak to someone” scenario https://cloud.google.com/support

Re:Tell HN: Google Cloud suspended our production projects a... by asojfdowgh on Monday August 22, 2022 @10:11AM (#32551622)
or just cluster over several other providers and make sure to test failover

the main problem with azure/aws/gcp is that their parent companies will track everything you and others do, and then close every related thing because they have enough market share to not give a fuck about you.

There are more than enough blog posts of enterprise customers running into random issues with the big thing, buying "big three, promise edition" isn't much of an assurance.

I'd suspect most of those customers have rapid migration plans written up. For a reason.

Re:Tell HN: Google Cloud suspended our production projects a... by mh- on Monday August 22, 2022 @10:19AM (#32551762)
> I'd suspect most of those customers have rapid migration plans written up.

I can assure you from experience that most large customers do not. For either major cloud, most large customers will have adopted proprietary services like AWS Redshift or GCP's Cloud Bigtable or Bigquery. None of these have anything like the possibility of a "rapid migration".

Re:Tell HN: Google Cloud suspended our production projects a... by fphhotchips on Monday August 22, 2022 @09:43PM (#32559583)
I'm not sure GCP really comes up in discussions about "either major cloud". Distant third place, looking more like becoming fourth before they get to second.
Re:Tell HN: Google Cloud suspended our production projects a... by mhoad on Monday August 22, 2022 @03:22PM (#32555976)
I feel like you’ve just ignored everything I mentioned, made up a bunch of hand wavy stuff and ended up with what might be actively the worst idea (multi-cloud) as a solution.

Not sure what to say in response here…

Re:Tell HN: Google Cloud suspended our production projects a... by SoftTalker on Monday August 22, 2022 @12:15PM (#32553542)
As I mentioned elsewhere, this is a sign of imbalance in the relationship. If you are too small to matter to Google or even to your account rep at Google, then you should not be using them.
Re:Tell HN: Google Cloud suspended our production projects a... by conductr on Monday August 22, 2022 @01:13PM (#32554320)
It's well known that Google does bare minimum customer support, was that considered during vendor selection?
Re:Tell HN: Google Cloud suspended our production projects a... by itsoktocry on Monday August 22, 2022 @10:19AM (#32551753)
>How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway?

I mean, it's an error.

Sure, it's infuriating, and in worst-case could financially harm (or cripple) your business. But Google didn't intend for it to happen. That makes no sense from a business perspective.

So, as others here suggest, you have to plan around the fact that mistakes like this can happen, especially when you're small.

Re:Tell HN: Google Cloud suspended our production projects a... by dafelst on Monday August 22, 2022 @10:31AM (#32551939)
I think the difficulty is (ESPECIALLY with Google and suspected fraud) that when they make a mistake, getting it corrected is basically impossible unless you know a person on the inside like an established account manager or a highly ranked exec.
Re:Tell HN: Google Cloud suspended our production projects a... by bastawhiz on Tuesday August 23, 2022 @12:54AM (#32560607)
I'd be inclined to think it's simply an error if people didn't return to HN to post strikingly similar stories month after month. Meanwhile, I rarely hear of similar stories with AWS or Azure or IBM or any other similar service.
Re:Tell HN: Google Cloud suspended our production projects a... by latexr on Monday August 22, 2022 @07:53AM (#32550136)
According to the OP, Google assured them a suspension wouldn’t happen then did it anyway.

In the car analogy it’s akin to looking the driver in the eyes and them signalling to you it’s safe to cross, only to hit the gas and run you over as you do. No amount of looking both ways before crossing would have saved you.

Re:Tell HN: Google Cloud suspended our production projects a... by jquery on Monday August 22, 2022 @06:07AM (#32549433)
It sounds more like they looked both ways, they saw a car, they made eye contact and the driver gestured for them to cross, then when they began to cross the driver ran them over. And now they’re getting advice to never cross streets as a pedestrian. Sound advice can also be victim blaming.
Re:Tell HN: Google Cloud suspended our production projects a... by Nebasuke on Monday August 22, 2022 @06:50AM (#32549708)
I got hit by a car in a situation similar to this. They made eye contact (I thought), were slowing down, and still hit me anyway by accelerating, despite me having the right of way on a bike. Turns out it was the first time they used the car, 1 minute into the drive and their focus was somewhere else.

I would say the advice here is, wait until the car is basically stopped before you cross the street rather than trusting it will because you think all the signs say they will. I do think this is solid advice for the production systems as well. All the signs can point to yes, but making extra assurances to avoid the bad path is valid for something critical like your production systems (or your life).

Re:Tell HN: Google Cloud suspended our production projects a... by black_puppydog on Monday August 22, 2022 @07:13AM (#32549858)
> wait until the car is basically stopped

you mean like an email literally saying they acknowledge the situation and won't suspend your account?

Re:Tell HN: Google Cloud suspended our production projects a... by shyn3 on Monday August 22, 2022 @07:56AM (#32550152)
In my experience, when dealing with vendors having an account manager can fix these kind of problems and prevent them.

Most places outsource their support to global teams don't have access. If you have an AM , in many places, they receive the same notice you receive, also they will personally send it and address it to you with a reminder. They also have a lot more sway as the sales side of every company has the power so they can get around all the process issues.

As the other commenter posted above, if you can have an account manager where you have critical services, it will provide you safety at the cost of a higher spend or minimum spending amounts.

Edit: Had about $1mm in GCP spend and couldn't open a ticket to get windows running properly. It took a week to find the AM and a day after he had it fixed.

Re:Tell HN: Google Cloud suspended our production projects a... by omniglottal on Monday August 22, 2022 @05:56PM (#32557804)
Paying extra is not a viable solution for not receiving what one has already paid for.
Re:Tell HN: Google Cloud suspended our production projects a... by Nebasuke on Monday August 22, 2022 @07:51AM (#32550124)
I think an email from support is more like "confirmation that they checked whether the car has seen me".

I'm not sure how successfully you've dealt with support with companies, but I have almost never been able to rely on the promises of a support agent, as they are simply not the ones actually making the decisions (in this case on whether your account gets suspended or not).

Re:Tell HN: Google Cloud suspended our production projects a... by scrollaway on Monday August 22, 2022 @10:11AM (#32551624)
A support agent is supposed to speak on behalf of the company. If they say "your account will not be suspended for this issue", and the company then suspends the account for that issue, that should be grounds to sue.
Re:Tell HN: Google Cloud suspended our production projects a... by Nebasuke on Tuesday August 23, 2022 @05:26AM (#32562371)
Sure, I don't disagree and it's silly/illegal when it happens, but this doesn't mean you can't protect yourself (sometimes without much more effort) to prevent damage or even just more effort. I think most people have spoken to a support agent who said they'd cancel a phone/broadband contract, or worse, had to do something slightly off script and then confirmed that the agreed XYZ would happen?

Regularly XYZ does not happen, or not exactly in the way you agreed. They cancel your broadband too late, or your internet starts a few weeks too late, etc. You can try to get compensation, or even sue, but some of the damage has already been done at that point. While this okay for temporarily overpaying broadband, this is probably not the case for a company's main production system, where the risk is very high. The company might not even survive to sue.

Re:Tell HN: Google Cloud suspended our production projects a... by black_puppydog on Monday August 22, 2022 @11:18AM (#32552679)
sounds more like you want a written and signed letter from the driver that they don't intend to restart the car, co-signed by their spouse
Re:Tell HN: Google Cloud suspended our production projects a... by ix101 on Monday August 22, 2022 @08:14AM (#32550270)
If a driver gestures to cross and hits the brakes and the brakes don't work it's hard to blame the driver if they didn't design the car. Just saying it might be more of a technical issue and the behaviour probably wasn't normal/expected.
Re:Tell HN: Google Cloud suspended our production projects a... by omniglottal on Monday August 22, 2022 @05:59PM (#32557842)
In this situation, the driver is at fault and everyone can sue the manufacturer/mechanic. Back to Google: they are at fault and responsible for damages, period. What benefit is there in differentiating a company from an employee authorized to communicate and act on their behalf?
Re:Tell HN: Google Cloud suspended our production projects a... by Hnrobert42 on Monday August 22, 2022 @05:05AM (#32549035)
I don’t take GP’s comment that way. I take it as a suggestion for how to avoid issues. I try not to be quick to judge when someone is offering helpful information.
Re:Tell HN: Google Cloud suspended our production projects a... by helsinkiandrew on Monday August 22, 2022 @05:02AM (#32549015)
Hang on - what would you do if you were running a business with hundreds of thousands of users and someone doesn't pay? Was it an accident? has the customer gone bust, or died, or are they trying to defraud you and get a month or two of free service.

Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they don't go through - but not suitable unless you're going to check that payments are made, which the OP seems not to have done. Perhaps GCP notifications should be better (I've experience the similar scenario with AWS and everything got fixed within a day) - but surely its up to the customer to make sure they pay.

Re:Tell HN: Google Cloud suspended our production projects a... by Beltalowda on Monday August 22, 2022 @06:35AM (#32549603)
I actually had some payment troubles with Linode a few months back; I had cancelled one of my bank accounts and forgot the CC there was tied to Linode, and because credit cards are not very common where I live and it can take a bit to request one so I couldn't "just" pay. Oops!

100% a mistake on my part of course; I emailed Linode, explained the above, and they said "sure, is 3 weeks time enough to sort it out? I've delayed any action on your account until such-and-such date but let us know if you need more time".

I've been a Linode customer for several years, paying about $100/month. Just a guy running some small things, hardly a big customer.

If Linode can do it, then surely Google can. Yes, you will probably have the occasional non-payment, but is kicking your people off your platform for a simple administrative mistake/mixup that much better? Accepting the occasional non-payment is better than kicking off people at the drop of a hat. I've since paid several more $100/month to Linode and everyone won in this situation: Linode because they kept a customer who will keep paying, and me because I could get my stuff sorted without having to worry about having my services cancelled.

Google's policy is simply short-sighted on every level, and harms Google's business interests too. The reason they can do things like this is because they have money to spare. Every business that's not swimming in cash like Scrooge McDuck treats their customers like this; the "big tech" company are the exceptions.

Re:Tell HN: Google Cloud suspended our production projects a... by newaccount74 on Monday August 22, 2022 @07:54AM (#32550140)
Linode stopped my services because of 20$ outstanding balance that I ignored for two weeks (was on vacation). When I tried to update the payment method, the card wasn't accepted, and after a few tries I got an error message that I made too many attempts and wouldn't be allowed to change it for the next 24h.

Fortunately they have customer support, and 5 minutes after emailing them they resumed my services again.

Re:Tell HN: Google Cloud suspended our production projects a... by htrp on Monday August 22, 2022 @08:23AM (#32550342)
Exactly!

On GCP you're basically SOL

Re:Tell HN: Google Cloud suspended our production projects a... by darkwater on Monday August 22, 2022 @07:21AM (#32549916)
> If Linode can do it, then surely Google can.

No, because Linode is using humans to interact with you. Google will not humans in those situation, their mantra is to automate everything so they can scale up with minimal human effort, even when it doesn't make sense to scale up that way.

Re:Tell HN: Google Cloud suspended our production projects a... by Beltalowda on Monday August 22, 2022 @07:38AM (#32550038)
That's just a choice Google made. They can do it like Linode, they just choose not to.
Re:Tell HN: Google Cloud suspended our production projects a... by mcv on Monday August 22, 2022 @08:25AM (#32550365)
I hope they can also automatically scale down when their customers flee to the competition.
Re:Tell HN: Google Cloud suspended our production projects a... by LegitShady on Monday August 22, 2022 @08:05AM (#32550207)
oh it makes sense for them, it just doesnt make sense if youre a customer.
Re:Tell HN: Google Cloud suspended our production projects a... by nosianu on Monday August 22, 2022 @05:13AM (#32549098)
Your rhetorical questions don't match OPs story. According to OP they did pay, manually, not even by credit card.

> however we went ahead and made a manual payment covering all the outstanding amount + extra

(the entire last paragraph of OPs text has those and more details)

I suggest not using and then arguing with and against fantasized stuff when we are talking about a concrete problem for which we have a concrete description, no?

Given that OP paid, of what use is your objection about non-paying customers here?

You also omit this important part:

> They ... assured the project will not be suspended.

Re:Tell HN: Google Cloud suspended our production projects a... by helsinkiandrew on Monday August 22, 2022 @07:03AM (#32549799)
> Your rhetorical questions don't match OPs story. According to OP they did pay, manually, not even by credit card.

That's fair enough - somehow I completely misread/understood that part of the post.

But my main point was that you should never be using credit cards for services that you rely on - particularly when billing/payment is automated and hidden from you. Credit card payments and checks fail regularly (10%+ for recurring payment charges) for legitimate (you don't have funds) and less legitimate reasons (a banks AI system thinking the charge looks fraudulent due to something outside of your account etc). Vendors are likely to stop the service as soon as they can get away with so they don't loose money.

Re:Tell HN: Google Cloud suspended our production projects a... by snowwrestler on Monday August 22, 2022 @08:33AM (#32550436)
A “manual” payment on GCP is usually still by credit card. You just set the amount by hand and hit submit, instead of the scheduled auto-bill.
Re:Tell HN: Google Cloud suspended our production projects a... by sbuk on Monday August 22, 2022 @06:14AM (#32549475)
It's simple - you give them notice during local working hours (i.e. not the weekend), that way the customer has time to rectify, and if they don't you can take action and have proof of warning them of the consequences of ignoring the notification. If they are having payment issues, it affords both parties time to discuss the issues and see what can reasonably done. It's called customer service - something that Google clearly struggles with.
Re:Tell HN: Google Cloud suspended our production projects a... by bastawhiz on Monday August 22, 2022 @06:58AM (#32549770)
I run a business with thousands of users. When payment fails, I try again eight times over two weeks. You get notices each time. There's no surprises. There's no excuse for shutting someone off abruptly for credit card issues.
Re:Tell HN: Google Cloud suspended our production projects a... by the__alchemist on Monday August 22, 2022 @10:30AM (#32551924)
I strongly disagree. The host must reach out to the customer, person-to-person, before shutting off a service.
Re:Tell HN: Google Cloud suspended our production projects a... by soheil on Monday August 22, 2022 @12:02PM (#32553351)
This is the line where HN becomes not so useful. Maybe try to imagine there are other professions beyond engineering that have existed in parallel that have their way doing things oh I don't know like following standards and best practices in accounting.
Re:Tell HN: Google Cloud suspended our production projects a... by jasonlotito on Monday August 22, 2022 @09:24AM (#32550997)
If you are providing someone a service, and you rely on a third party to provide that service, and that third party screws you over, then by extension, you are screwing over your customers.

That is your fault, and your customers have every right to blame you. You are responsible for the due diligence on which providers you use.

Blaming a 3rd party might absolve you of some of the blame, but your still not providing the service you agreed to provide.

And before you try to suggest anything else, the condition here was "having customers you can't let down."

You have customers you can't let down. That's the context.

Re:Tell HN: Google Cloud suspended our production projects a... by michaelcampbell on Tuesday August 23, 2022 @08:18AM (#32563734)
I'll counterargue with a saying from the driving context; "The graveyard is full of people who were `right`."
[dead]
Re:Tell HN: Google Cloud suspended our production projects a... by baobabKoodaa on Monday August 22, 2022 @04:51AM (#32548932)
For the record, I was being sarcastic and do not actually believe it was the customer's fault. I believe this is an abhorrent way of running business by Google.
Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @05:41AM (#32549288)
Right, but I think your sarcasm was misplaced. It is actually possible for customers to make a mistake. These mistakes are relatively easy to prevent. It could be that in this particular case there wasn't anything the OP could have done and they are simply unlucky. But that's very unlikely.
Re:Tell HN: Google Cloud suspended our production projects a... by TomSwirly on Monday August 22, 2022 @05:59AM (#32549382)
All - absolutely all - the evidence we have seems to show that Google was completely wrong - that a payment didn't go off because of a known Google outage, that the company then made a manual payment, but started to receive threatening messages, that they appealed to billing who told them they were safe, but then the site was cut off in the middle of the night on the weekend.

Your argument appears to be this: "Facts are unimportant: the customer is almost certainly wrong in every case."

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @06:48AM (#32549686)
Epistimological check-in: we don't actually have any evidence, except for a very high level story from the OP without much relevant detail. So all we have is the base rate, which is in almost all cases when there are billing problems they are at least to some extent because of negligence or fraud on the part of the customer.
Re:Tell HN: Google Cloud suspended our production projects a... by yjftsjthsd-h on Monday August 22, 2022 @08:57AM (#32550685)
The base rate of building problems is that the customer screwed up, and the base rate of Google support problems is that Google is negligent. I'm not sure how to compare these.
Re:Tell HN: Google Cloud suspended our production projects a... by paskozdilar on Monday August 22, 2022 @06:03AM (#32549412)
> These mistakes are relatively easy to prevent.

Assuming one knows about them. How would one learn about them, without making the same mistakes themselves?

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @06:45AM (#32549667)
Most of it will actually be documented by your provider, and is often pretty obvious. For example, if you're paying with card, have at least one extra as backup payment method (which most providers allow and recommend). For extra safety: make one of these cards a pre-paid debit card loaded with enough money to pay for at least one month. The one thing you don't want to happen is to miss a payment, which could happen for many reasons that aren't under your full control, like a technical problem with one of the cards, a fraud detection false positive rendering your card invalid, etc. And once you are unable to complete a payment, that's when the trouble begins.
Re:Tell HN: Google Cloud suspended our production projects a... by acka on Monday August 22, 2022 @10:06AM (#32551545)
intellectronica:

> For example, if you're paying with card, have at least one extra as backup payment method (which most providers allow and recommend).

Please don't generalize. There are many countries where credit cards aren't common, and applying for just one (and getting it approved) can already involve a lot of work and take lots of time. Such as: the bank and the national credit authority needing to do a thorough credit check, you having to supply proof of identity and stable income, the bank having to register your new line of credit with the national credit authority, etcetera. Applying for multiple CC's, if at all possoble, only adds more trouble: your registered total amount of credit will rise (even if you haven't used any of it yet) which can limit your options for any other loans like a mortgage, a car loan, a subscription, and so on.

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @10:38AM (#32552071)
That's true - it's a rather privileged view to assume that everyone all over the world enjoys the same facilities, I know that. If you are somewhere where you can't have services people from rich countries consider basic, like credit cards, then you are left out. That sucks. Hopefully the more decentralized internet being built these days will be a solution.
Re:Tell HN: Google Cloud suspended our production projects a... by eastbound on Monday August 22, 2022 @02:56PM (#32555630)
No.

When the due invoice represents less than 5% of your lifetime value, the least they can do is eat the cost and wait for you to resolve the payment, because, at worst, they’ll lose 5%.

Re:Tell HN: Google Cloud suspended our production projects a... by baobabKoodaa on Monday August 22, 2022 @05:48AM (#32549336)
I have taken your advice and will keep this in mind when dealing with cloud providers in the future. That said, I still place the blame on cloud providers when things like this happen.
Re:Tell HN: Google Cloud suspended our production projects a... by rb666 on Monday August 22, 2022 @05:25AM (#32549189)
And then you get hit by the car, even though the light was green. Doesn't help you, does it?
Re:Tell HN: Google Cloud suspended our production projects a... by baobabKoodaa on Monday August 22, 2022 @05:29AM (#32549214)
More like: the driver stops and gestures me to walk over the road. As I step in front of the car, they suddenly jump on the gaspedal and drive over me.
Re:Tell HN: Google Cloud suspended our production projects a... by Aeolun on Monday August 22, 2022 @06:34AM (#32549593)
Using Google is kinda like trying to cross a 10 lane highway without getting hit.
Re:Tell HN: Google Cloud suspended our production projects a... by e4325f on Monday August 22, 2022 @04:52AM (#32548945)
whoosh