News for hackers. Stuff that matters.
Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @04:30AM (#32548782)
Generally, all cloud providers have two ways of engaging with and billing customers. One is using a credit card, which is very easy for getting started. The other one is having an established relationship with a sales / account team and setting up billing by invoice. Smaller companies sometimes don't understand that and stick with credit card billing for too long. Once you move to production and have your own customers that you can't let down, you should really make the extra effort and move to invoice billing (with all the upfront overhead that entails).
Re:Tell HN: Google Cloud suspended our production projects a... by baobabKoodaa on Monday August 22, 2022 @04:46AM (#32548901)
Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.
Re:Tell HN: Google Cloud suspended our production projects a... by doix on Monday August 22, 2022 @05:06AM (#32549055)
It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car.

Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.

Re:Tell HN: Google Cloud suspended our production projects a... by bastawhiz on Monday August 22, 2022 @06:53AM (#32549739)
How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway? Having a relationship with them doesn't matter if they still can't be trusted to do business. The type of payment method you use is not a factor that plays into whether you can trust your vendor.
Re:Tell HN: Google Cloud suspended our production projects a... by mcv on Monday August 22, 2022 @08:19AM (#32550315)
I don't think you can. My takeaway from this discussion is definitely that GCP is only suitable for toy projects, and not for serious production stuff that requires continuity.
Re:Tell HN: Google Cloud suspended our production projects a... by mhoad on Monday August 22, 2022 @08:58AM (#32550696)
This is a fundamentally unserious point unless you want to add more qualifiers.

I’m pointing to this as evidence https://cloud.google.com/customers

I feel like a lot of the problems that people run into with GCP are avoided with two main components.

Number one you should look seriously at the “enterprise” designation for different products and services if you are looking for long term stability and guarantees about the future (details: https://cloud.google.com/apis/docs/resources/enterprise-apis) and secondly you need to buy actual customer support as an add on if you want to avoid the “I had to hit the front page of HN to actually speak to someone” scenario https://cloud.google.com/support

Re:Tell HN: Google Cloud suspended our production projects a... by asojfdowgh on Monday August 22, 2022 @10:11AM (#32551622)
or just cluster over several other providers and make sure to test failover

the main problem with azure/aws/gcp is that their parent companies will track everything you and others do, and then close every related thing because they have enough market share to not give a fuck about you.

There are more than enough blog posts of enterprise customers running into random issues with the big thing, buying "big three, promise edition" isn't much of an assurance.

I'd suspect most of those customers have rapid migration plans written up. For a reason.

Re:Tell HN: Google Cloud suspended our production projects a... by mh- on Monday August 22, 2022 @10:19AM (#32551762)
> I'd suspect most of those customers have rapid migration plans written up.

I can assure you from experience that most large customers do not. For either major cloud, most large customers will have adopted proprietary services like AWS Redshift or GCP's Cloud Bigtable or Bigquery. None of these have anything like the possibility of a "rapid migration".

Re:Tell HN: Google Cloud suspended our production projects a... by fphhotchips on Monday August 22, 2022 @09:43PM (#32559583)
I'm not sure GCP really comes up in discussions about "either major cloud". Distant third place, looking more like becoming fourth before they get to second.
Re:Tell HN: Google Cloud suspended our production projects a... by mhoad on Monday August 22, 2022 @03:22PM (#32555976)
I feel like you’ve just ignored everything I mentioned, made up a bunch of hand wavy stuff and ended up with what might be actively the worst idea (multi-cloud) as a solution.

Not sure what to say in response here…

Re:Tell HN: Google Cloud suspended our production projects a... by SoftTalker on Monday August 22, 2022 @12:15PM (#32553542)
As I mentioned elsewhere, this is a sign of imbalance in the relationship. If you are too small to matter to Google or even to your account rep at Google, then you should not be using them.
Re:Tell HN: Google Cloud suspended our production projects a... by conductr on Monday August 22, 2022 @01:13PM (#32554320)
It's well known that Google does bare minimum customer support, was that considered during vendor selection?
Re:Tell HN: Google Cloud suspended our production projects a... by itsoktocry on Monday August 22, 2022 @10:19AM (#32551753)
>How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway?

I mean, it's an error.

Sure, it's infuriating, and in worst-case could financially harm (or cripple) your business. But Google didn't intend for it to happen. That makes no sense from a business perspective.

So, as others here suggest, you have to plan around the fact that mistakes like this can happen, especially when you're small.

Re:Tell HN: Google Cloud suspended our production projects a... by dafelst on Monday August 22, 2022 @10:31AM (#32551939)
I think the difficulty is (ESPECIALLY with Google and suspected fraud) that when they make a mistake, getting it corrected is basically impossible unless you know a person on the inside like an established account manager or a highly ranked exec.
Re:Tell HN: Google Cloud suspended our production projects a... by bastawhiz on Tuesday August 23, 2022 @12:54AM (#32560607)
I'd be inclined to think it's simply an error if people didn't return to HN to post strikingly similar stories month after month. Meanwhile, I rarely hear of similar stories with AWS or Azure or IBM or any other similar service.
Re:Tell HN: Google Cloud suspended our production projects a... by latexr on Monday August 22, 2022 @07:53AM (#32550136)
According to the OP, Google assured them a suspension wouldn’t happen then did it anyway.

In the car analogy it’s akin to looking the driver in the eyes and them signalling to you it’s safe to cross, only to hit the gas and run you over as you do. No amount of looking both ways before crossing would have saved you.

Re:Tell HN: Google Cloud suspended our production projects a... by jquery on Monday August 22, 2022 @06:07AM (#32549433)
It sounds more like they looked both ways, they saw a car, they made eye contact and the driver gestured for them to cross, then when they began to cross the driver ran them over. And now they’re getting advice to never cross streets as a pedestrian. Sound advice can also be victim blaming.
Re:Tell HN: Google Cloud suspended our production projects a... by Nebasuke on Monday August 22, 2022 @06:50AM (#32549708)
I got hit by a car in a situation similar to this. They made eye contact (I thought), were slowing down, and still hit me anyway by accelerating, despite me having the right of way on a bike. Turns out it was the first time they used the car, 1 minute into the drive and their focus was somewhere else.

I would say the advice here is, wait until the car is basically stopped before you cross the street rather than trusting it will because you think all the signs say they will. I do think this is solid advice for the production systems as well. All the signs can point to yes, but making extra assurances to avoid the bad path is valid for something critical like your production systems (or your life).

Re:Tell HN: Google Cloud suspended our production projects a... by black_puppydog on Monday August 22, 2022 @07:13AM (#32549858)
> wait until the car is basically stopped

you mean like an email literally saying they acknowledge the situation and won't suspend your account?

Re:Tell HN: Google Cloud suspended our production projects a... by shyn3 on Monday August 22, 2022 @07:56AM (#32550152)
In my experience, when dealing with vendors having an account manager can fix these kind of problems and prevent them.

Most places outsource their support to global teams don't have access. If you have an AM , in many places, they receive the same notice you receive, also they will personally send it and address it to you with a reminder. They also have a lot more sway as the sales side of every company has the power so they can get around all the process issues.

As the other commenter posted above, if you can have an account manager where you have critical services, it will provide you safety at the cost of a higher spend or minimum spending amounts.

Edit: Had about $1mm in GCP spend and couldn't open a ticket to get windows running properly. It took a week to find the AM and a day after he had it fixed.

Re:Tell HN: Google Cloud suspended our production projects a... by omniglottal on Monday August 22, 2022 @05:56PM (#32557804)
Paying extra is not a viable solution for not receiving what one has already paid for.
Re:Tell HN: Google Cloud suspended our production projects a... by Nebasuke on Monday August 22, 2022 @07:51AM (#32550124)
I think an email from support is more like "confirmation that they checked whether the car has seen me".

I'm not sure how successfully you've dealt with support with companies, but I have almost never been able to rely on the promises of a support agent, as they are simply not the ones actually making the decisions (in this case on whether your account gets suspended or not).

Re:Tell HN: Google Cloud suspended our production projects a... by scrollaway on Monday August 22, 2022 @10:11AM (#32551624)
A support agent is supposed to speak on behalf of the company. If they say "your account will not be suspended for this issue", and the company then suspends the account for that issue, that should be grounds to sue.
Re:Tell HN: Google Cloud suspended our production projects a... by Nebasuke on Tuesday August 23, 2022 @05:26AM (#32562371)
Sure, I don't disagree and it's silly/illegal when it happens, but this doesn't mean you can't protect yourself (sometimes without much more effort) to prevent damage or even just more effort. I think most people have spoken to a support agent who said they'd cancel a phone/broadband contract, or worse, had to do something slightly off script and then confirmed that the agreed XYZ would happen?

Regularly XYZ does not happen, or not exactly in the way you agreed. They cancel your broadband too late, or your internet starts a few weeks too late, etc. You can try to get compensation, or even sue, but some of the damage has already been done at that point. While this okay for temporarily overpaying broadband, this is probably not the case for a company's main production system, where the risk is very high. The company might not even survive to sue.

Re:Tell HN: Google Cloud suspended our production projects a... by black_puppydog on Monday August 22, 2022 @11:18AM (#32552679)
sounds more like you want a written and signed letter from the driver that they don't intend to restart the car, co-signed by their spouse
Re:Tell HN: Google Cloud suspended our production projects a... by ix101 on Monday August 22, 2022 @08:14AM (#32550270)
If a driver gestures to cross and hits the brakes and the brakes don't work it's hard to blame the driver if they didn't design the car. Just saying it might be more of a technical issue and the behaviour probably wasn't normal/expected.
Re:Tell HN: Google Cloud suspended our production projects a... by omniglottal on Monday August 22, 2022 @05:59PM (#32557842)
In this situation, the driver is at fault and everyone can sue the manufacturer/mechanic. Back to Google: they are at fault and responsible for damages, period. What benefit is there in differentiating a company from an employee authorized to communicate and act on their behalf?
Re:Tell HN: Google Cloud suspended our production projects a... by Hnrobert42 on Monday August 22, 2022 @05:05AM (#32549035)
I don’t take GP’s comment that way. I take it as a suggestion for how to avoid issues. I try not to be quick to judge when someone is offering helpful information.
Re:Tell HN: Google Cloud suspended our production projects a... by helsinkiandrew on Monday August 22, 2022 @05:02AM (#32549015)
Hang on - what would you do if you were running a business with hundreds of thousands of users and someone doesn't pay? Was it an accident? has the customer gone bust, or died, or are they trying to defraud you and get a month or two of free service.

Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they don't go through - but not suitable unless you're going to check that payments are made, which the OP seems not to have done. Perhaps GCP notifications should be better (I've experience the similar scenario with AWS and everything got fixed within a day) - but surely its up to the customer to make sure they pay.

Re:Tell HN: Google Cloud suspended our production projects a... by Beltalowda on Monday August 22, 2022 @06:35AM (#32549603)
I actually had some payment troubles with Linode a few months back; I had cancelled one of my bank accounts and forgot the CC there was tied to Linode, and because credit cards are not very common where I live and it can take a bit to request one so I couldn't "just" pay. Oops!

100% a mistake on my part of course; I emailed Linode, explained the above, and they said "sure, is 3 weeks time enough to sort it out? I've delayed any action on your account until such-and-such date but let us know if you need more time".

I've been a Linode customer for several years, paying about $100/month. Just a guy running some small things, hardly a big customer.

If Linode can do it, then surely Google can. Yes, you will probably have the occasional non-payment, but is kicking your people off your platform for a simple administrative mistake/mixup that much better? Accepting the occasional non-payment is better than kicking off people at the drop of a hat. I've since paid several more $100/month to Linode and everyone won in this situation: Linode because they kept a customer who will keep paying, and me because I could get my stuff sorted without having to worry about having my services cancelled.

Google's policy is simply short-sighted on every level, and harms Google's business interests too. The reason they can do things like this is because they have money to spare. Every business that's not swimming in cash like Scrooge McDuck treats their customers like this; the "big tech" company are the exceptions.

Re:Tell HN: Google Cloud suspended our production projects a... by newaccount74 on Monday August 22, 2022 @07:54AM (#32550140)
Linode stopped my services because of 20$ outstanding balance that I ignored for two weeks (was on vacation). When I tried to update the payment method, the card wasn't accepted, and after a few tries I got an error message that I made too many attempts and wouldn't be allowed to change it for the next 24h.

Fortunately they have customer support, and 5 minutes after emailing them they resumed my services again.

Re:Tell HN: Google Cloud suspended our production projects a... by htrp on Monday August 22, 2022 @08:23AM (#32550342)
Exactly!

On GCP you're basically SOL

Re:Tell HN: Google Cloud suspended our production projects a... by darkwater on Monday August 22, 2022 @07:21AM (#32549916)
> If Linode can do it, then surely Google can.

No, because Linode is using humans to interact with you. Google will not humans in those situation, their mantra is to automate everything so they can scale up with minimal human effort, even when it doesn't make sense to scale up that way.

Re:Tell HN: Google Cloud suspended our production projects a... by Beltalowda on Monday August 22, 2022 @07:38AM (#32550038)
That's just a choice Google made. They can do it like Linode, they just choose not to.
Re:Tell HN: Google Cloud suspended our production projects a... by mcv on Monday August 22, 2022 @08:25AM (#32550365)
I hope they can also automatically scale down when their customers flee to the competition.
Re:Tell HN: Google Cloud suspended our production projects a... by LegitShady on Monday August 22, 2022 @08:05AM (#32550207)
oh it makes sense for them, it just doesnt make sense if youre a customer.
Re:Tell HN: Google Cloud suspended our production projects a... by nosianu on Monday August 22, 2022 @05:13AM (#32549098)
Your rhetorical questions don't match OPs story. According to OP they did pay, manually, not even by credit card.

> however we went ahead and made a manual payment covering all the outstanding amount + extra

(the entire last paragraph of OPs text has those and more details)

I suggest not using and then arguing with and against fantasized stuff when we are talking about a concrete problem for which we have a concrete description, no?

Given that OP paid, of what use is your objection about non-paying customers here?

You also omit this important part:

> They ... assured the project will not be suspended.

Re:Tell HN: Google Cloud suspended our production projects a... by helsinkiandrew on Monday August 22, 2022 @07:03AM (#32549799)
> Your rhetorical questions don't match OPs story. According to OP they did pay, manually, not even by credit card.

That's fair enough - somehow I completely misread/understood that part of the post.

But my main point was that you should never be using credit cards for services that you rely on - particularly when billing/payment is automated and hidden from you. Credit card payments and checks fail regularly (10%+ for recurring payment charges) for legitimate (you don't have funds) and less legitimate reasons (a banks AI system thinking the charge looks fraudulent due to something outside of your account etc). Vendors are likely to stop the service as soon as they can get away with so they don't loose money.

Re:Tell HN: Google Cloud suspended our production projects a... by snowwrestler on Monday August 22, 2022 @08:33AM (#32550436)
A “manual” payment on GCP is usually still by credit card. You just set the amount by hand and hit submit, instead of the scheduled auto-bill.
Re:Tell HN: Google Cloud suspended our production projects a... by sbuk on Monday August 22, 2022 @06:14AM (#32549475)
It's simple - you give them notice during local working hours (i.e. not the weekend), that way the customer has time to rectify, and if they don't you can take action and have proof of warning them of the consequences of ignoring the notification. If they are having payment issues, it affords both parties time to discuss the issues and see what can reasonably done. It's called customer service - something that Google clearly struggles with.
Re:Tell HN: Google Cloud suspended our production projects a... by bastawhiz on Monday August 22, 2022 @06:58AM (#32549770)
I run a business with thousands of users. When payment fails, I try again eight times over two weeks. You get notices each time. There's no surprises. There's no excuse for shutting someone off abruptly for credit card issues.
Re:Tell HN: Google Cloud suspended our production projects a... by the__alchemist on Monday August 22, 2022 @10:30AM (#32551924)
I strongly disagree. The host must reach out to the customer, person-to-person, before shutting off a service.
Re:Tell HN: Google Cloud suspended our production projects a... by soheil on Monday August 22, 2022 @12:02PM (#32553351)
This is the line where HN becomes not so useful. Maybe try to imagine there are other professions beyond engineering that have existed in parallel that have their way doing things oh I don't know like following standards and best practices in accounting.
Re:Tell HN: Google Cloud suspended our production projects a... by jasonlotito on Monday August 22, 2022 @09:24AM (#32550997)
If you are providing someone a service, and you rely on a third party to provide that service, and that third party screws you over, then by extension, you are screwing over your customers.

That is your fault, and your customers have every right to blame you. You are responsible for the due diligence on which providers you use.

Blaming a 3rd party might absolve you of some of the blame, but your still not providing the service you agreed to provide.

And before you try to suggest anything else, the condition here was "having customers you can't let down."

You have customers you can't let down. That's the context.

Re:Tell HN: Google Cloud suspended our production projects a... by michaelcampbell on Tuesday August 23, 2022 @08:18AM (#32563734)
I'll counterargue with a saying from the driving context; "The graveyard is full of people who were `right`."
[dead]
Re:Tell HN: Google Cloud suspended our production projects a... by baobabKoodaa on Monday August 22, 2022 @04:51AM (#32548932)
For the record, I was being sarcastic and do not actually believe it was the customer's fault. I believe this is an abhorrent way of running business by Google.
Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @05:41AM (#32549288)
Right, but I think your sarcasm was misplaced. It is actually possible for customers to make a mistake. These mistakes are relatively easy to prevent. It could be that in this particular case there wasn't anything the OP could have done and they are simply unlucky. But that's very unlikely.
Re:Tell HN: Google Cloud suspended our production projects a... by TomSwirly on Monday August 22, 2022 @05:59AM (#32549382)
All - absolutely all - the evidence we have seems to show that Google was completely wrong - that a payment didn't go off because of a known Google outage, that the company then made a manual payment, but started to receive threatening messages, that they appealed to billing who told them they were safe, but then the site was cut off in the middle of the night on the weekend.

Your argument appears to be this: "Facts are unimportant: the customer is almost certainly wrong in every case."

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @06:48AM (#32549686)
Epistimological check-in: we don't actually have any evidence, except for a very high level story from the OP without much relevant detail. So all we have is the base rate, which is in almost all cases when there are billing problems they are at least to some extent because of negligence or fraud on the part of the customer.
Re:Tell HN: Google Cloud suspended our production projects a... by yjftsjthsd-h on Monday August 22, 2022 @08:57AM (#32550685)
The base rate of building problems is that the customer screwed up, and the base rate of Google support problems is that Google is negligent. I'm not sure how to compare these.
Re:Tell HN: Google Cloud suspended our production projects a... by paskozdilar on Monday August 22, 2022 @06:03AM (#32549412)
> These mistakes are relatively easy to prevent.

Assuming one knows about them. How would one learn about them, without making the same mistakes themselves?

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @06:45AM (#32549667)
Most of it will actually be documented by your provider, and is often pretty obvious. For example, if you're paying with card, have at least one extra as backup payment method (which most providers allow and recommend). For extra safety: make one of these cards a pre-paid debit card loaded with enough money to pay for at least one month. The one thing you don't want to happen is to miss a payment, which could happen for many reasons that aren't under your full control, like a technical problem with one of the cards, a fraud detection false positive rendering your card invalid, etc. And once you are unable to complete a payment, that's when the trouble begins.
Re:Tell HN: Google Cloud suspended our production projects a... by acka on Monday August 22, 2022 @10:06AM (#32551545)
intellectronica:

> For example, if you're paying with card, have at least one extra as backup payment method (which most providers allow and recommend).

Please don't generalize. There are many countries where credit cards aren't common, and applying for just one (and getting it approved) can already involve a lot of work and take lots of time. Such as: the bank and the national credit authority needing to do a thorough credit check, you having to supply proof of identity and stable income, the bank having to register your new line of credit with the national credit authority, etcetera. Applying for multiple CC's, if at all possoble, only adds more trouble: your registered total amount of credit will rise (even if you haven't used any of it yet) which can limit your options for any other loans like a mortgage, a car loan, a subscription, and so on.

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @10:38AM (#32552071)
That's true - it's a rather privileged view to assume that everyone all over the world enjoys the same facilities, I know that. If you are somewhere where you can't have services people from rich countries consider basic, like credit cards, then you are left out. That sucks. Hopefully the more decentralized internet being built these days will be a solution.
Re:Tell HN: Google Cloud suspended our production projects a... by eastbound on Monday August 22, 2022 @02:56PM (#32555630)
No.

When the due invoice represents less than 5% of your lifetime value, the least they can do is eat the cost and wait for you to resolve the payment, because, at worst, they’ll lose 5%.

Re:Tell HN: Google Cloud suspended our production projects a... by baobabKoodaa on Monday August 22, 2022 @05:48AM (#32549336)
I have taken your advice and will keep this in mind when dealing with cloud providers in the future. That said, I still place the blame on cloud providers when things like this happen.
Re:Tell HN: Google Cloud suspended our production projects a... by rb666 on Monday August 22, 2022 @05:25AM (#32549189)
And then you get hit by the car, even though the light was green. Doesn't help you, does it?
Re:Tell HN: Google Cloud suspended our production projects a... by baobabKoodaa on Monday August 22, 2022 @05:29AM (#32549214)
More like: the driver stops and gestures me to walk over the road. As I step in front of the car, they suddenly jump on the gaspedal and drive over me.
Re:Tell HN: Google Cloud suspended our production projects a... by Aeolun on Monday August 22, 2022 @06:34AM (#32549593)
Using Google is kinda like trying to cross a 10 lane highway without getting hit.
Re:Tell HN: Google Cloud suspended our production projects a... by e4325f on Monday August 22, 2022 @04:52AM (#32548945)
whoosh
Re:Tell HN: Google Cloud suspended our production projects a... by klohto on Monday August 22, 2022 @04:51AM (#32548936)
What is this fud? Both AWS and Azure won't suspend your account for several months, even with charges failing. Especially not if you're communicating with them. It's incredibly easy to get a hold of human support in case of the other clouds.
Re:Tell HN: Google Cloud suspended our production projects a... by jaywalk on Monday August 22, 2022 @05:18AM (#32549134)
My company isn't very big (we spend in the very low five-figures with Azure per month) and yet we were assigned an "Azure Success Manager" (or something close to that) who I have a standing monthly phone call with, and can reach out to at any time to oversee any issues that may come up. We actually did have a billing issue come up a while ago (it was Microsoft's fault) and although our account wasn't in any real danger of being shut off, she stayed on top of the case and made sure it got resolved quickly.
Re:Tell HN: Google Cloud suspended our production projects a... by wongarsu on Monday August 22, 2022 @08:32AM (#32550423)
> we spend in the very low five-figures with Azure per month

That's over $100,000 a year, at that point having a Key Account Manger (or Success Manager or whatever) to talk to should be the expectation.

Re:Tell HN: Google Cloud suspended our production projects a... by jaywalk on Monday August 22, 2022 @12:35PM (#32553809)
I agree. But although I've never used GCP, I get the feeling that an account of my size would not get the same treatment.
Re:Tell HN: Google Cloud suspended our production projects a... by htrp on Monday August 22, 2022 @08:24AM (#32550354)
That's because Microsoft understands and hires humans for customer support.....
Re:Tell HN: Google Cloud suspended our production projects a... by RA_Fisher on Monday August 22, 2022 @07:46AM (#32550096)
That’s awesome customer service! Thanks for sharing.
Re:Tell HN: Google Cloud suspended our production projects a... by nucleardog on Monday August 22, 2022 @09:55AM (#32551399)
Had AWS payment issues at one company spending ~$20k/mo. Had some technical issues trying to catch the bill up. Contacted support, obviously concerned. It was entirely a non-event to them. Basically just said as long as you're in contact with us, we're not gonna shut anything off don't worry about it. And... they didn't.

At other companies where we've had an active and involved account rep literally just shot off one email and got back a "no worries, I put a block on the account so it can't be suspended without my approval".

I know in the past I've had personal accounts go a couple months in the red when a card expired and I wasn't staying on top of the emails.

These sorts of things are really only an issue with one specific provider.

Re:Tell HN: Google Cloud suspended our production projects a... by CSDude on Monday August 22, 2022 @06:28AM (#32549559)
I forgot to pay 200$ Azure bill on a test account for 3 months because I missed the mails. They just waived that one after 3 months without me even requesting and I've been paying since. Google is just hostile to everyone by default.
Re:Tell HN: Google Cloud suspended our production projects a... by JacobiX on Monday August 22, 2022 @06:30AM (#32549571)
We use Azure and Linode for all of our cloud services. The number one reason: we can contact human support at any time and we have a designated customer success manager. In many situation this proved to be critical, especially for resolving paiement issues, etc
Re:Tell HN: Google Cloud suspended our production projects a... by laegooose on Monday August 22, 2022 @06:32AM (#32549586)
yes, in my experience, AWS keeps running everything as is when there is a problem with the payment method or a disagreement about bill
Re:Tell HN: Google Cloud suspended our production projects a... by andix on Monday August 22, 2022 @07:44AM (#32550070)
Ofc they don’t. Because they charge A LOT for the service. Even if the chance of the customer paying is just 30% it’s probably still a profit (it’s mostly licensing costs you pay with cloud bills, infrastructure costs are maybe 20% of it; hardware, electricity, cooling). And on the way they can charge a lot of late fees, that raises profit even more.
Re:Tell HN: Google Cloud suspended our production projects a... by belter on Monday August 22, 2022 @04:54AM (#32548969)
These type of stories, that appear regularly on HN, seem in the vast majority, related to only one specific provider.
Re:Tell HN: Google Cloud suspended our production projects a... by bbarnett on Monday August 22, 2022 @05:00AM (#32549002)
Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison.

I really can't understand why anyone would use any of their services. Support counts. It's one of the first things I look at, when evaluating any external partner. What does it cost, how long does it take for it to happen, etc?

Whatever anyone else says in this thread, that's what you need to determine for yourself. Whatever company, what is the support channel like, how responsive is it?

If you don't know? If you can't get immediate emergency response and support, and I mean immediate, then why are you even hosting there?

Re:Tell HN: Google Cloud suspended our production projects a... by markbnj on Monday August 22, 2022 @09:25AM (#32551002)
> Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison.

We have been running production on Google Cloud for six years. We are not a large customer: our spend is in the low five figures. We pay for a middle-tier support plan and support has been excellent during this entire period. We very rarely need to take advantage of it, since the underlying systems are extremely stable and almost never cause us any issues (compute engine, GKE, GCS, Memorystore, Firebase, CloudSQL, etc). Just thought casual readers of this thread could use a counterpoint.

Re:Tell HN: Google Cloud suspended our production projects a... by herdcall on Monday August 22, 2022 @11:17AM (#32552652)
Good for you, but if you used the service only "very rarely," I don't think you've experienced the problems that are being discussed here. I had signed up for paid support for a project a while back and remember having so much trouble even paying for it (some trouble with invoices), let alone get any value out of it, that I ended up canceling it.
Re:Tell HN: Google Cloud suspended our production projects a... by davidspiess on Monday August 22, 2022 @10:05AM (#32551532)
Exactly our experience too. We are extremely happy with the stability and quality of their services.
Re:Tell HN: Google Cloud suspended our production projects a... by _fat_santa on Monday August 22, 2022 @09:39AM (#32551192)
> I really can't understand why anyone would use any of their services.

I can speak to this because a few months ago when I started a project, I went with Firebase. At the time the rationale was, I need something more complex than Netlify/Vercel, but not AWS that will suck up all my time figuring out IAM roles and other dumb stuff.

It helped speed things up in the prototyping stage, but I will likely move my infra over to AWS soon.

Re:Tell HN: Google Cloud suspended our production projects a... by vbezhenar on Monday August 22, 2022 @05:36AM (#32549262)
Google tech is the best. Google invented k8s, their k8s cloud is the best in the world. There's nothing similar to GKE Autopilot. Similar things could be said about some other services. People want the best cloud and they hope that those issues won't affect them.
Re:Tell HN: Google Cloud suspended our production projects a... by jhugo on Monday August 22, 2022 @06:17AM (#32549496)
We're heavy k8s users at work. We used to use GKE, now we use a mixture of EKS and bare metal after migrating most stuff away from GCP. Honestly, GKE isn't better than k8s anywhere else in any meaningful way at all.
Re:Tell HN: Google Cloud suspended our production projects a... by unity1001 on Monday August 22, 2022 @05:50AM (#32549349)
You can just use default K8 without any added service. It works well enough. There is no obligation to use GC for K8. There are a lot of good base K8 providers.
Re:Tell HN: Google Cloud suspended our production projects a... by valenterry on Monday August 22, 2022 @09:04AM (#32550752)
Google has some good tech and then most part of the cloud is just years behind. Want an example: you can't configure the gcp docker registry [EDIT: artifact registry] to cleanup images automatically (after a number of days or number of images or anything).

This is total standard in AWS and the ticket for this on GCP has been open for many years now.

Re:Tell HN: Google Cloud suspended our production projects a... by derefr on Monday August 22, 2022 @09:47AM (#32551294)
Isn’t GCR backed by a GCS bucket in your own project? Can’t you configure object lifecycle management on said GCS bucket?
Re:Tell HN: Google Cloud suspended our production projects a... by valenterry on Monday August 22, 2022 @10:14AM (#32551667)
GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion)

That is not backed by GCS buckets - if it were, I'm sure that I would have found this solution while searching the web.

But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.

Re:Tell HN: Google Cloud suspended our production projects a... by derefr on Monday August 22, 2022 @11:07AM (#32552522)
> But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.

https://cloud.google.com/storage/docs/lifecycle#numberofnewe...

> GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion)

You're right that this functionality seems to not be built into the Artifact Registry backend (and that's weird†), but it does still exist: see https://github.com/GoogleCloudPlatform/gcr-cleaner (found linked from https://cloud.google.com/artifact-registry/docs/docker/manag...), and specifically the `keep` flag for it.

Note also how the project is hosted under the GoogleCloudPlatform GH org. To me, when I see GCP projects that are set up like this (in the GH org but disclaimed as "not official" in the GCP docs), this suggests that Google engineers built it knowing it's a pain point; and those engineers will support it to the best of their ability in the capacity of being maintainers of this open-source project; but Google as a company don't want to officially support it (yet), and so your GCP support contract won't get you any business-level support for it.

It's sort of like how, in Postgres, there is code which is maintained by the Postgres maintainers, but which lives under contrib/ as an extension. It's essentially a lability-waiver for that component.

---

† I do have a guess as to why Google do things this way. At least where dev tools are concerned, Google seems to eschew the usual distributed-systems architecture for long-running jobs (of having a thin API client binary that submits jobs to a cloud-side control-plane daemon, which then drives the job forward, and which can then be polled/subscribed for job status by said client.) Rather, Google seemingly have a philosophy of designing local fat clients that reach into the cloud to drive backend processes as the "control node" for those processes. The Cloud Dataflow (⇒ Apache Beam) architecture is designed this way, for example. I believe it's the reason that the Google Cloud SDK ships with so many binaries — there are a lot of fat clients in there that actually drive logic, rather than just sending messages to daemons that drive the logic.

And, presuming developers are issued good workstations, I can see the advantages of this architecture. A local control node synchronously knows its own status, rather than having to poll for it; a local control node can use local resources (like how Cloud Dataflow can consume and produce local files on the ends of the pipeline with the same streaming efficiency as a regular CLI text-processing shell command); and an operation started by mistake, with local control, can be cancelled by just ctrl+c-ing the control process.

Depending on how you design the client, it can also "mandate manual usage" — i.e. ensure that the developer is interactively running the process for the process to proceed, and therefore that said dev is available in case anything goes wrong. (I've personally dropped [async daemon-driven] Continuous Deployment, in favor of this sort of "synchronous dev-workstation-driven deployment.")

I wish someone at Google would write up a paper on this philosophy; it's pretty clearly implicit in a lot of their work, but I've never seen it mentioned explicitly anywhere. (Maybe it's just one dev-tools lead who has influenced a lot of these projects, doing what they think is "obvious"?)

Re:Tell HN: Google Cloud suspended our production projects a... by valenterry on Monday August 22, 2022 @02:42PM (#32555454)
> https://cloud.google.com/storage/docs/lifecycle#numberofnewe...

That is object specific though, so probably won't work unless I use exactly one image.

> derefr 3 hours ago | parent | context | flag | on: Tell HN: Google Cloud suspended our production pro...

> But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.

https://cloud.google.com/storage/docs/lifecycle#numberofnewe...

> > GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion) > > You're right that this functionality seems to not be built into the Artifact Registry backend (and that's weird†), but it does still exist: see https://github.com/GoogleCloudPlatform/gcr-cleaner (found linked from https://cloud.google.com/artifact-registry/docs/docker/manag...), and specifically the `keep` flag for it.

No, it does not "exist". It's either builtin and then it exists, or it doesn't. What you linked is a way for me to build it myself. And I found this AND this solution is even linked in the years old Google ticket. And guess what, they didn't build it. On AWS no problem. As I said: years behind.

Sure, I can setup my own cron job (or here cloud run function / github action). But that's not what I expect from a leading cloud vendor. This is not a niche feature!

Please don't defend it, Google doesn't deserve it. Credits to whoever build the 3rd party solution, but Google really failed here.

> Depending on how you design the client, it can also "mandate manual usage" — i.e. ensure that the developer is interactively running the process for the process to proceed

Just no.

Let's face it: Google is just too incompetent to do it. Not the developers there, but the company as a whole in the way it is organized. Even IF it were as you said and it would be a philosophy, they could say that close the ticket, but it's still open.

I can list you dozens of similar things with GCP and related Google services.

Re:Tell HN: Google Cloud suspended our production projects a... by billythemaniam on Monday August 22, 2022 @08:12AM (#32550256)
Yeah I agree overall. BigQuery, Spanner, Cloud Run, Pub/Sub, Vertex AI. Like all clouds, some of the services suck, like Data Fusion, but overall the services work well and work well together. In particular, the way Google Cloud makes authentication and permissions seamless is amazing. Google does struggle with support though.
Re:Tell HN: Google Cloud suspended our production projects a... by andix on Monday August 22, 2022 @07:49AM (#32550116)
K8S runs well on all major cloud providers. You can run it on premise without licensing costs quite well (for example k3s, rancher, …). Or fully managed with Open Shift.

AWS, Azure and GC are all on the same level. Any of them gives you „the best“ technology.

Re:Tell HN: Google Cloud suspended our production projects a... by vbezhenar on Monday August 22, 2022 @11:45AM (#32553110)
https://www.reddit.com/r/kubernetes/comments/wuboxb/best_kub...

This is recent topic and you can see that so many people prefer GKE. That's not my opinion.

Sure, any cloud works good enough.

Re:Tell HN: Google Cloud suspended our production projects a... by bavell on Monday August 22, 2022 @09:39AM (#32551199)
Last time I used AKS (~1yr ago) it was a joke compared to GKE.
Re:Tell HN: Google Cloud suspended our production projects a... by htrp on Monday August 22, 2022 @08:26AM (#32550368)
> Google tech is the best.

True, but unfortunately they are literally the worst at customer service.

Re:Tell HN: Google Cloud suspended our production projects a... by RajT88 on Monday August 22, 2022 @10:39AM (#32552101)
A fellow I know used to do contract work with GCP. Same horror stories.
Re:Tell HN: Google Cloud suspended our production projects a... by gnfargbl on Monday August 22, 2022 @07:12AM (#32549854)
It would be nice if Google actually told their customers that this is an option. As a result of your post, I realise that I quality for invoiced billing and have just applied for it. Thanks!

If I had an "established relationship with a sales / account team" then yeah, perhaps I would have been informed. Unfortunately my experience is that unless you're spending above some relatively high threshold (>$10k/mo? >$100k/mo? >$1m/mo?) then Google don't prioritise your business.

Re:Tell HN: Google Cloud suspended our production projects a... by TomSwirly on Monday August 22, 2022 @05:56AM (#32549372)
> Smaller companies sometimes don't understand that [...] you should really make the extra effort

Let me get this straight. This company does everything they were supposed to, paid on time, and Google's own error cuts them off, and you are blaming them?!?

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @06:53AM (#32549732)
You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't.
Re:Tell HN: Google Cloud suspended our production projects a... by Cederfjard on Monday August 22, 2022 @07:57AM (#32550158)
I think it’s fair not to take any given story at face value, without ”receipts” to back it up. You however seem to me to go in the other direction and barely want to give OP the benefit of the doubt. What details would you like to see in order to be convinced?

From your profile I clicked through to your LinkedIn via your website, and I think it would’ve been becoming of you to disclose that you were a senior engineering manager at Google Cloud up until earlier this year, given the context of the discussion and your stance on it.

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @08:04AM (#32550199)
And now I work for the competition :) In any case, I know enough about how cloud services work to be able to comment intelligently, and don't have any interest in Google (except for some stock I hold indirectly via ETFs). This is all just common sense plus what I know from having worked for several cloud services providers.
Re:Tell HN: Google Cloud suspended our production projects a... by rfrey on Monday August 22, 2022 @08:41AM (#32550502)
If you assume, without evidence, that an HN user is lying if they share a story outside your experience -- there is little reason to use HN.
Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @09:25AM (#32551015)
I don't understand why you think that I assume anyone is lying. Nothing I said suggests that.
Re:Tell HN: Google Cloud suspended our production projects a... by rfrey on Monday August 22, 2022 @12:54PM (#32554058)
The original poster said they paid their bill, they were assured they would not have their account suspended, then it was suspended anyhow despite them being fully paid up.

Your parent said OP did what they were supposed to and paid, but got cut off anyhow -- to which you replied "You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't."

I'm not sure how to interpret that other than you saying OP is probably not being truthful in their account of the situation.

Re:Tell HN: Google Cloud suspended our production projects a... by dvfjsdhgfv on Monday August 22, 2022 @04:47AM (#32548907)
I'm very happy Hetzner has all these options built in from the very start. There's no need to make any "extra effort". Actually, most of my servers are on yearly billing to make things even easier for everybody.
Re:Tell HN: Google Cloud suspended our production projects a... by madeofpalk on Monday August 22, 2022 @04:55AM (#32548975)
It's not really the payment method that's key here, it's having a relationship with sales / account manager.
Re:Tell HN: Google Cloud suspended our production projects a... by smileybarry on Monday August 22, 2022 @06:34AM (#32549600)
Right, but Hetzner also just doesn’t auto-charge you (from my experience, at least). You get invoiced and pay it manually (there’s a link in the invoice), then the invoice is settled immediately in your account panel. I’ve even been late once by forgetting about it and they didn’t cut me off in any way.
Re:Tell HN: Google Cloud suspended our production projects a... by majewsky on Monday August 22, 2022 @07:51AM (#32550125)
> Hetzner also just doesn’t auto-charge you (from my experience, at least).

I have my Hetzner account set up so they pull the money automatically (through SEPA Direct Debit). Not sure how it works with credit cards, but it's not like pull payments are not possible with Hetzner.

Re:Tell HN: Google Cloud suspended our production projects a... by smileybarry on Monday August 22, 2022 @08:30AM (#32550405)
Ah, I see. I use PayPal and while PP supports payment agreements (I have that setup with a few sites), Hetzner insists I pay each invoice manually. Maybe just bank debit payments support "auto pulling".
Re:Tell HN: Google Cloud suspended our production projects a... by tmpz22 on Monday August 22, 2022 @01:02PM (#32554174)
Can we talk about how google monetizes (weaponizes) this sales relationship? They force it on you as "common sense" but it makes them A LOT of money.

Ultimately its as bad as car dealerships to require a special connection to get basic tasks completed.

Re:Tell HN: Google Cloud suspended our production projects a... by FBISurveillance on Monday August 22, 2022 @04:57AM (#32548984)
Except finance.

At certain scale (in my case, 5 figures/mo spend with a vendor and up), paying upfront is not something finance usually likes and NET-60/NET-90 is the norm.

Re:Tell HN: Google Cloud suspended our production projects a... by dvfjsdhgfv on Monday August 22, 2022 @05:30AM (#32549229)
Actually, it was exactly finance in the case of one of my clients. "Dear Mr dvfjsdhgfv, we're receiving invoices from this Hetzner company for some ridiculously small amounts each month which adds to our overhead, could you kindly ask them to bill us yearly? Thanks!"
Re:Tell HN: Google Cloud suspended our production projects a... by cinntaile on Monday August 22, 2022 @06:22AM (#32549521)
It's quite unfortunate that your parents decided to call you dvfjsdhgfv.
Re:Tell HN: Google Cloud suspended our production projects a... by smileybarry on Monday August 22, 2022 @06:36AM (#32549612)
“Mr. Dvfjsdhgfv is my father. Please, call me Shcyahudd Dvfjsdhgfv”
Re:Tell HN: Google Cloud suspended our production projects a... by FBISurveillance on Monday August 22, 2022 @07:15AM (#32549878)
Just passing feedback from my peers at the surveillance van: fantastic joke.
Re:Tell HN: Google Cloud suspended our production projects a... by cosmodisk on Monday August 22, 2022 @06:05AM (#32549422)
Of course they don't like it, but that's why CTOs and the likes should be talking about these things in management meetings and not just fairy dust future plans.
Re:Tell HN: Google Cloud suspended our production projects a... by junon on Monday August 22, 2022 @04:53AM (#32548960)
Someone just the other day said Hetzner did the exact same thing though...
Re:Tell HN: Google Cloud suspended our production projects a... by dvfjsdhgfv on Monday August 22, 2022 @05:28AM (#32549204)
I'm not a blind fan of Hetzner and I'm aware of their limitations, but billing works extremely well for me, and they're very patient with me when I forget to pay an invoice (sending friendly reminders and not threatening to cut me off right away).

However, I remember a case when I was the culprit - spam was being sent from one of my servers (one of the users had a common username and password, something I should have never allowed) and Hetzner notified me immediately, but they didn't block the server in any way (something I'd expect). They gave me some window for explanation/solving the problem. I was very happy with how they handled the issue - but of course others may have had different experiences.

Re:Tell HN: Google Cloud suspended our production projects a... by oefrha on Monday August 22, 2022 @06:17AM (#32549497)
> billing works extremely well for me

I suppose billing works extremely well for most GCP customers as well.

As for me, I hosted a side project on Hetzner a few years back for some European presence. IIRC I used PayPal for payment and they didn't support PayPal recurring payment, so I had to settle an invoice each month. Used them for about two years without problem, then missed payment for two months during a chaotic period when I didn't notice their monthly invoice emails. I got exactly one email reminder about missed payment titled "Reminder" (yes, that's all, I got all their correspondence in front of me right now) before the final "Cancellation of Contract" email and termination of my account. At that point, politely pleading with them to reinstate the account was useless, I only got a threatening letter back demanding a wire transfer of the ~€60 I owed within three business days or risk being sent to collections.

Was I in the wrong? Sure. Were they friendly or patient? No.

Re:Tell HN: Google Cloud suspended our production projects a... by aestetix on Monday August 22, 2022 @06:25AM (#32549537)
Hetzner also has a customer service phone number that you can call. I've never seen anything like that from Google.
Re:Tell HN: Google Cloud suspended our production projects a... by fabioyy on Monday August 22, 2022 @05:41AM (#32549290)
i was BANNED from hetzner when i forgot to Pay ( it has been almost 10 years ).at the time i offer to Pay any debts, but they said that i was not welcome anymore.
Re:Tell HN: Google Cloud suspended our production projects a... by harha on Monday August 22, 2022 @04:55AM (#32548974)
If you’re dealing with customers that way you should have a big red warning sign everywhere to not run anything in production under any circumstances.

Otherwise don’t take their money and advertise that everything will be easier and more reliable

Re:Tell HN: Google Cloud suspended our production projects a... by thedougd on Monday August 22, 2022 @07:25AM (#32549954)
Cloud providers have a minimum spend for invoice billing. For GCP you need to spend at least $2500 a month for 3 months to qualify.
Re:Tell HN: Google Cloud suspended our production projects a... by raxxorraxor on Monday August 22, 2022 @10:56AM (#32552363)
AWS seems to be patient with that. Had the wrong credit card information from a customer once. Credit card validation seems to take a while and only fails a day later or so. Put in the wrong info a few times and invoices were already piling up since I only checked once a week or so. Probably still a good tip to move to a better payment scheme.
Re:Tell HN: Google Cloud suspended our production projects a... by patchtopic on Monday August 22, 2022 @10:26AM (#32551864)
the small azure project that got canned by Azure I was working on, credit card was the only way to pay, and Azure suspended my employers Azure account with no explanation why (though it appears in hindsight to be credit card related), but they refused to explain why. this victim blaming gaslighting is ridiculous.
Re:Tell HN: Google Cloud suspended our production projects a... by XCabbage on Monday August 22, 2022 @09:25AM (#32551007)
How would invoice billing even reduce the risk of this outcome? Surely any system with more manual and asynchronous steps involved is MORE likely to result in Google erroneously claiming you haven't paid, not less? I don't grasp what the logic behind this suggestion is even meant to be.
Re:Tell HN: Google Cloud suspended our production projects a... by derefr on Monday August 22, 2022 @09:52AM (#32551354)
Invoices are indefinitely postpaid; there’s no real point at which you can really 100% declare an invoice as having “not been paid.”

That’s why NET30/90/etc exist — without an incentive to pay sooner, invoices will just sit around on the receiver’s books until it’s convenient for them to pay (e.g. to make their financials for a particular quarter look good.)

Invoices under contract law are like PayPal’s arbitration process: “always favours the buyer.” A seller would need an explicit pre-existing contract clause stating a due date for an invoice, in order for a due date declared on an invoice to have any force; by default, it doesn’t. And because invoice billing is something companies tend to offer mostly to companies they’re trying to stay on the good side of, they don’t tend to sit them down for a binding contract negotiation when doing so.

Re:Tell HN: Google Cloud suspended our production projects a... by intellectronica on Monday August 22, 2022 @09:36AM (#32551155)
The invoice itself doesn't solve anything, but it requires a "handshake" step where trust is established with the payer, with the result that things don't just turn off automatically when something goes wrong.
Re:Tell HN: Google Cloud suspended our production projects a... by croes on Monday August 22, 2022 @06:39AM (#32549634)
And then an account error happens and your server goes down.

I don't think OP's post is about payment options but about problem handling.