Worst case, if AWS or Google decide to fuck you over, let the bill bounce. This way you've still got funds on hand to deal with the fall out and some leverage to negotiate a solution.
The problem (in the article at least) isn't being surreptitiously billed a large amount, it is cost of a critical service changing dramatically without warning.
Also, nothing gets them talking to you faster than a bounced invoice :)
It's when a big bill lands in error or is introduced sneakily that having a separate "cloud only card" can save you. In that case, it's the supplier that's gone nuclear and you're just responding in kind.
Sorry, but this is the risk of using somebody else's service -- especially Google's -- as a critical component of a business. It may be marvellous for developers not to bother with infrastructure for their own service, but this strategy likes to come back and bite in the ass.
I only wish setting up virtual cards were as easy as setting up a new mail box. I know there are services out there that lets you do this, but I've yet to find one that works globally and is easy to work with.
The second bank account is really more for protecting you against black swan events. If all your accounts are under the same bank and have the same owner, there's nothing stopping the bank taking funds from one of your accounts to make up for shortfalls in another.
The only reason I don't use them for everything is because I get better cashback/points with other cards.
I'm wary of giving my card information to companies outside India because I don't have this protection.
True for Indian companies but what if you get charged via Forex abroad? For me - Google or Steam - don't required 2FA. All I have to do is to enter my number, confirm CVV then zap. It gets debited. I do get a call from my bank (HDFC) asking if this transaction has been initiated by me, if I say yes, they let go.
The Flash UI looks like this http://mayoyo.tokyo/jJ7.png
They used to have a mobile app but it was discontinued since it didn't support 2FA.
> What countries are you available in?
> We're only available in the United States at the moment.
> Is Emburse available outside of the U.S.?
> Only companies incorporated within the U.S. may register for an Emburse account. In addition, the primary account owner must be a resident of the United States with a Social Security number and a verifiable physical U.S. street address (no P.O. Boxes). We do not offer Emburse accounts to residents of Puerto Rico, Guam, the U.S. Virgin Islands, American Samoa, and other U.S. territories.
But Paypal, for instance, can only empty a bank account that is used for nothing else, any service like this (open-ended liabilities with shitty/no support) is isolated to debit cards, etc. Nobody gets permission to charge open-ended amounts to credit cards, and nobody gets automation rights to the "real" bank accounts, period.
There's a business model here somewhere for someone to automate all of this. It involves substantially more time to keep organized than my prior two-banks/two-credit cards model. Personal finance apps just aren't designed to treat, for instance, debit cards as transient.
As others note, sure, services that do something like this can still come after you. But who has the money in their pocket right now matters a lot when resolving disputes like this.
After all, they're just being helpful in managing your money and paying your suppliers.
I thought what they were doing is applying all your debits, charging you if that puts you into overdraft, and then applying any credits. If they applied the credits first, you wouldn't go below zero.
Either way, you're trying to charge $46 against an account with $20, but one method results in 1 fee and one method results in 3 fees. The issue of credits vs debits you mentioned can compound this issue.
Consider a bank account with $20 in it, the day before payday, when you forget your World of Warcraft subscription hits today instead of tomorrow. That's $15. But you buy:
* Lunch: $5
* Coffee: $2
* Dinner: $5
* A late-night snack: $5
If the transactions happened in-order, the subscription causes the overdraft, and you're charged $25 or whatever (yeah they are very high). But in the way banks used to do it, they'd take your total ledger for the day, and order it by highest $ first. So, subscription first, $5 left. Lunch, $0 left. Coffee, dinner, and snack are three transactions when overdrawn, so that's $25 * 3 = $75 in fees instead.
Banks claimed that this process was because you'd probably want your highest-amounts paid first, as it'd be something like rent money. In the end, it means way more fees for them, though.
$75 in fees is especially brutal when you don't have much money in the first place; it pretty much caused me to not be able to even go to 7-11 for a candy bar for two months.
I'd advise not using debit cards for anything, under any circumstances, and relying on the better protections that credit cards provide, but it's a losing battle. My debit card is ATM-only.
This is generally a good approach for all sorts of reasons, not just protection from over enthusiastic cloud bills.
Keep most of your funds in one bank (and never use the cards). Transfer from this account to one or more other accounts with other banks and use the cards on these secondary accounts to pay for things. This way you're protected from suppliers over billing and banks going crazy with unarranged overdrafts (which aren't secured so absolute worst case you just tell them to fuck off and default on it).
Basically, the trick is holding on to your cash. Much harder to fix things rapidly if you're having to negotiate with a third party between you and the supplier (which is the case with a credit card company).
Edit: just remembered we're using Google's Cloud Vision API and that's billed to the same debit card. So, for UK accounts at least, they take debit cards.
EDIT: I have no affiliation with privacy.com; I'm just a satisfied user.
I don't understand why the larger internet/tech community keeps giving Google a pass on this. Anything where money comes in or out should ultimately have a support line where ultimately a human customer service agent can respond.
Shout far and wide (twitter, facebook, reddit, etc) that [Big corporation] is screwing you. If you can give it some sort of spin - like racism or gender equality - all the better. Just kick up as much of a storm as you can.
I guarantee that [Big corporation] will be forced to respond.
(I say this semi-jokingly, but it is sad that this a) works, and b) that sometimes it is the only way to get their attention)
Last week at thegoodfellas.com.br a disgruntled ex-employee managed to erase all their Gmail accounts and change the administrator account in order to prevent recovery. You can recover a deleted account in 5 days, but you need access to that administrator account. Unfortunately they could not recover access to the administrator account in time and lost all their G Suite data - you may blame them for their flawed firing process but shit happens and when it does a reseller can be fast enough to prevent a tragedy.
Disclaimer: I run a G Suite reseller operation
G Suite has incredible phone support. I've called 3-4 times over past 5 years and never waited more than 30 seconds to talk to a knowledgeable and capable tech who resolved the issue immediately. No BS troubleshooting steps, just straight to the heart of the issue.
I recall a couple Irish guys, one Eastern European and one Indian, but none with too-strong to parse accents (these were their locations -- relevant to make the point that Google clearly does have multiple live phone support centers around the world). All fully willing and capable of resolving the issue without passing me around. These weren't incredibly complex issues, but at least a couple were glitches requiring fixes on the backend (vs. mistakes on my part).
Has anyone with a G Suite account and a Cloud account ever tried contacting G Suite Support for the cloud side? Might be worth a try -- they've been helpful for me in one case where the issue wasn't strictly related to G Suite.
A local reseller can pay you a visit and verify the situation but hands are tied for the friendly Kumar sitting at a helpdesk facility in India.
Here in Brazil it is even harder, because most customers are unable to communicate well in English and support in Portuguese is not done by native speakers.
Pretty simple: opt-in KYC. Give people the option to email/fax you their passport or birth certificate or whatever, at any time after they set up their account but before the account is compromised. Extract the relevant ID numbers from the images; then store those numbers, encrypted, the same way you'd store "password recovery" info.
If the account is later compromised on their end, just ask the person attempting to do the recovery to send through the same stuff again, and compare with your recovery fields.
It's essentially the pseudo-biometric, pseudo-"something you have" equivalent of a Secret Question.
If you're a business, there's no excuse not to pay Google for support. It's $150/mo; significantly higher than the $25/mo infrastructure bill the OP was discussing, but not unreasonable if you want to have a chance of guaranteeing QoS to your customers.
Is there a certain size of vendor or price point where we expect to have human support and an SLA for response?
Yes, when they control the switch that can kill your business.
This is Google support summed up perfectly in a single sentence. It really is appalling.
Never quite understood what's "appalling" about it. They've never had strong support for any dev/cloud-oriented services. I know some folks who've had good experiences, but those were exceptions, not the rule.
I'll say, I get the frustration with it, but it's why I've never considered using their cloud services for anything. I had a client who asked me to migrate all their stuff to google 'cloud compute' and their 'sql cloud' offering last year. It was a performance disaster; queries that were minor .1ms on localhost or in the same network were suddenly 18-20ms hitting google cloud. Spent a few days trying to dig in, and a few responses from various groups were "shouldn't be that slow" and "works for me" and that was about it. Gave up trying to get any real response from google (we may have got a 'looking in to it' response - I don't remember).
Thing was, I knew going in it wasn't going to end well, I just wasn't sure exactly why (maybe that's just too cynical of a view, but I prefer 'realistic').
1. Things to work 2. People to answer when things don't work 3. Things to start working again in a timely manner, with timely and concise updates along the way
Points two and three carry a lot of nuance, but Google's support does not. It's a virtual wall that people cry to and get no response back from, except for the rare few occasions when the Google deities decide to grace forums or mailing lists with their presence. It's just not a serious offering.
Now, you can pay extra for support, but even so it's going to be appalling. You probably will get to talk to someone at least, but in my experience you rarely get ahold of anyone who'll actually read what you've sent them, much less actually try to help.
My take away is that support and customer service is just not a priority for Google, what so ever. It's not even a low priority thing, it's a no priority thing.
You know this by
a) hundreds and thousands of horror stories from people saying "I did not get any support from google". You can counter that with "here are people who did get support", but the existence of the thousands of support nightmare stories out there gives you enough evidence to know that this isn't really "supported" in the way you might expect.
b) there are no phone numbers. https://firebase.google.com/support/ - no phone numbers. That's, of course, not the sole criteria you should use to judge potential support, but... it's google.
It's 2017. We have multiple years of knowing what Google's track record is re: support for paying customers. Outside of large adwords advertisers (where they make a lot of money), it's non-existent from a practical standpoint.
The big problem here is that firebase was fine until google acquired them.
I'm reminded of a time when I had a bad experience at a restaurant. Local place I went to with friends from work. We went... probably 4-5 times per year, and I loved the chicken sandwich they did. Last time I went, it was ... bad. Not what I'd expected at all. It was different.
Server asks "how's everything?". I said I was disappointed that the chicken sandwich had changed - it was not as good as before. Now... I wasn't actually complaining as in "comp the meal", but I was not a satisfied diner.
"Nothing's changed sir".
"um... yeah - this is nowhere near as good as it used to be".
"You're wrong, nothing's changed".
We had a back and forth, and they went back and came out "the cooks say the recipe is exactly the same".
This was getting a bit ridiculous. Manager came over, same exchange. He leaves, the comes back and says "they're right - the recipe is 100% the same. We have changed a few ingredients last month, but it's the same".
Maybe that doesn't actually relate to the google service situation at all now, come to think of it, but I was reminded of that situation when I started typing it :)
Oh yeah - nothing changed from OP's end, except how the new owner (google) decided to start monitoring and accounting for usage. Think of a "$9.99 all you can eat buffet" getting loyal diners, then changing owners, and the new owners giving some people a $95 bill at the end of the meal because they'd decided to also charge for any visits to the salad bar over 2. Most diners would never hit that limit, but a few - loyal diners for years - would have been (rightly) upset at the after-the-fact change.
I know this but to say everyone does or should isn't exactly fair. As well, you assume I am holding the purse strings, that I'm in any way responsible or even have a say in procurement. This may be true in my own business (it is) but not necessarily elsewhere, like my clients (it more often than not isn't) and you'll just have to live with poor decisions made on your behalf.
Also quite often, there's a sunk cost that makes switching that much harder. So no, it's not fair to say "Google has shitty support but that's ok because we all kind of know this." Paying customers can and should complain about this, loudly, shouting it from the roof tops. Ideally they vote with their wallets and just leave Google, but again that's just easier said than done in many cases.
Some semblance of professionalism shouldn't be too much to ask from one of the world's biggest corporations, an ask coming from paying customers no less.
That's the ONLY thing that will ever make this change, but it rarely happens because "sunk cost" rationalizations (and "maybe it'll get better next year!").
Shouting from the rooftops for years has not worked. This is not anywhere near a 'new' problem.
Yes, possibly some people don't know it, and yeah, I totally get that "you'll just have to live with poor decisions made on your behalf.".
I live with poor decisions all the time (mine and others'). But the notion that we can bitch about it on forums, or make 'formal' requests for better support - none of that makes a lick of difference to them (or, demonstrably hasn't for several years). It's "google", so some people will flock to it because of the brand, or because it's not azure, or aws, or whatever. Or to tick the 'cloud' box on their spreadsheet. Or because google gave them thousands in free credits to switch.
"Some semblance of professionalism shouldn't be too much to ask from one of the world's biggest corporations, an ask coming from paying customers no less."
But... it demonstrably is, and has been.
And... you'll get a client who will say "I can't believe this, email someone, get this fixed". At some point you have to push back and say "no, this is simply not fixable, you have to live with XYZ, or pay the cost of moving somewhere else".
An extremely large customer, maybe, possibly, might have a bit of say, perhaps. But even if you're dropping millions with them... for a company that measures revenue in hundreds of millions of dollars per day, you probably won't hold much sway. If you had that sort of budget to spend, and it was that critical, you'd likely be doing enough due diligence that research in to actual support/service levels (vs what the sales contact tells you) would be on the agenda.
Vote with the wallet is the only viable/impactful approach, and people have to be willing to forgo the sunk costs.
And for people that don't know up front - they always had the opportunity to search (google) beforehand. If they choose not to - caveat emptor.
10 years ago, we could possibly be forgiven for choosing google's services (GAE, etc) because there wasn't as much of a track record to look at re: support, etc. There's no good excuse for starting a project today on google compute and claiming ignorance of their support levels.
If I've misrepresented what you're trying to say here please do correct me, because I clearly don't get it. If I haven't, I suppose we'll just have to agree to disagree – I think customers should shout from the rooftops till they're blue if they want to, even if it seems to lead nowhere.
But to avoid it, you generally do research ahead of time. And/or be willing to actually walk away from your 'investment' on their platform and start again somewhere else.
In the OP case, they didn't sign up with google in the first place. It's doubly shitty for them, because they didn't ask for this, and could not have known in advance that google would end up being the support provider. Similar to when I refinanced a house, and 6 month later, my 'servicer' sold my loan to someone else with much crappier service (charge me $3/month to take an electronic payment from me? I wouldn't have chosen this as a service option).
Yes, paying customers should go elsewhere. Complaining about it to google directly and indirectly for a decade has not noticeable improved their support for their cloud offerings.
Suck it up or go elsewhere. You can keep registering complaints with them, or organize digital protests, or whatever, but to get better service, you will likely have to go somewhere else.
"I think customers should shout from the rooftops till they're blue if they want to, even if it seems to lead nowhere". Of course they can. They just need to realize it will not change the quality of support. Someone earlier (you?) said to 'vote with your wallet'. Shouting from the digital rooftops about how crappy service X is, but continuing to pay them, is a waste of time at best, and enabling at worst. Leaving for another service is voting with your wallet/budget, and it needs to happen.
In the very short term, agreed - it doesn't do much good. But shouting from the rooftops internally (at client, management, stakeholders, etc) may do more good in the medium term.
Also, from the article, Firebase wasn't google when they went in, plus it was a side project that suddenly took off, not a thought-out business plan.
And yes, on the comic - ok, not everyone "knows" everything up front. But... you have search engines (google, even!) to do actual research beforehand.
I see small business folks and small consultancies do far more research on comparing various $7/month hosting plans via hosting review sites than I see people researching their decisions to use AWS vs Google vs Azure.
The above reasons are also why I don't use any open source software developed or maintained solely by Google. There's no guarantee that Google won't just stop maintaining it after a year or less. For a somewhat recent project, my co-founder and I evaluated using Angular2 for the front-end of our application. During our evaluation, we learned about Angular4. Knowing about what happened between Angular1 and Angular2, we decided to steer clear of the angular ecosystem. This decision also made more sense from the performance and ecosystem side of the argument. Most groups that develop open source software do their best to ensure that there is a well documented and simple upgrade path. Look at Django, rails, React, Vue, postgresql, etc for example. As far as I have experienced, Google products offer no such simple upgrade path.
To be honest, I'm not really sure what value Google is offering anymore. The only quality product from them I still find any use for is Gmail. Even Gmail has competitors that do Email better. I guess there's also youtube, but Google are kind of shitting that up as well.
I think the tech space needs to walk away from worshiping Google and Facebook. In all reality, the technical quality of their products is lacking and their entire goal is to lock both consumers and developers inside their own walled gardens.
But this is a somewhat easier choice to make because I still have the code locally. Google could stop supporting it tomorrow, and it would not be fun, but things wouldn't stop working, unlike the firebase situation above, where the option is "turn off your app".
That said, I've generally been using knockout and vue (more vue recently than knockout) and both have been fine.
"Third, they don't have any actual support at all for anything". There's a lot of people on this thread that feel like because they're paying for something they should get support. And... while I'm sympathetic to that viewpoint, and agree they should, we've had 10+ years of seeing google consistently under- or non- support many services that people seem to want to base their entire businesses on.
To be honest, if all of my Internet requests were 18 ms slower I wouldn't even notice.
if I read you correctly and it wasn't a typo, could you mention why you cared so much? The .1 ms you quoted on localhost is extraordinarily quick and I wouldn't expect it to translate to a cloud...
To (re?)clarify - this was 18ms per query to the SQL cloud (from a google compute engine).
Yes, were there things that could be done to reduce that? Of course. The other guy on the project put some caching in, reduced duplicate DB calls, etc, and got it down to around 3-4 seconds, but without severely diving in and learning far more about this inherited codebase, there wasn't much that could be done to make it much faster.
The other steps involved heavier caching, which changed the actual functionality of the service that was contracted before. "real time status updates" had a functional definition of being within X seconds, IIRC. Caching everything so 'new' information wasn't necessarily available to an end user for, say, 3 minutes, wasn't an option, without getting a lot of buy-in from a lot of people. Not saying it wouldn't work technically, but the client relationship was already strained.
The move to google was primarily to 'save money' (because someone had $10k service credit with google), but the effort to make it work well on google cloud and SQL combo (wasn't my call) was too much.
You can say this isn't google's fault entirely - you'd probably have a strong case - but not everyone is moving pristine clean/strong/good code over, and problems with the original base were amplified by seemingly small issues.
I hope you don't think I'm blaming your code base, but just out of pure curiosity, as an amateur question could you mention why the SQL calls couldn't be done in parallel -- did each one depend on the next or something?
At the high level I get everything you just said, just curious about this point, if you remember.
"What do you mean, you retain a running instance of every previous instance I pushed?!"
Fun times.
Support was somebody in a different timezone who complained when I didn't answer my phone in the middle of the night. I quite liked the Google Cloud offering, but to get so burnt so quickly... stay away.
We don't want to be calling customers in the middle of their night while they are sleeping. Its not an effective way to resolve issues. We need help identifying when we fall short so we can fix the issues.
Ways to let us know: 1) fill out the survey when your case is closed. We review every survey but less then 1% of cases get the survey filled out. 2) tweet your case id at us and it also goes through our review process 3) email me ([email protected]) and I can correct the problems. 4) GCP slack community has a whole bunch of active Googlers (and the community will flag one us down) if you are having problems with the support team.
What? Seriously? Is this documented anywhere?
Would you recommend against using GAE?
I've been burned so hard by them, I'd recommend against using anything Google.
My google usage is now
1. Occasionally watching Youtube videos in an incognito window
2. Very rarely using !g to search Google via DDG
3. Using Gmaps in an incognito window a couple times a month
and that's it. I don't find that I miss it at all. If I had to stop using Youtube and Gmaps, I won't miss 'em.
[0] https://serverfault.com/questions/218615/mail-server-checkli...
I'm not a big account with them, surely trivial compared to those mass domain parkers you mention, but they've never made me feel like I was less valued as a customer. I have received calls from humans just to ask me if I am currently satisfied with my service with them.
Especially if you have a business-critical service, this should simply be a baseline requirement for support, but it's becoming shockingly rare these days.
SPF is essentially a self-maintained whitelist.
Blacklists are public and searchable. Something to check after dealing with an infected machine, for example.
Ironically, commercial spammers have the best SPF records. And DKIM. And all the other goodies.
It was more of a mental "but but my decade-long history of gmail" thing, but it turned out that really was less of a big deal than I thought. I now have a nice custom domain and I forward my gmail to my fastmail, updating subscriptions every so often. Eventually gmail will receive no more mail I care about and I will remove the forwarding altogether. Its getting there!
Looking back on it now, about a year after migrating, my only regret is I didn't do it sooner.
If you aren't a FastMail customer yet, you're gonna assume we're shills. We're not, it's just the sort of quality product that makes you want to tell others. It's like having found Gmail in 2005.
No angels anywhere.
Now, if only good and valuable startups weren't getting bought by google by the dozens :( I guess it's your cue to start migrating to something in house when it happens.
So, for some applications, GCP works very well.
Yes, for that reason a POC I made for a side project costed almost 100 bucks to me (I was the single user)... I never checked the current bill because I was sure the total cost would have been around 15-20... when I got charged I noticed all the previous pushed versions had a single instance running, so the costs were incremental...
It may be my fault for not adequately reading all their caveats, but it seems to me that the default experience shouldn't be like that.
I love Google, but their Cloud Platform is still incomprehensible to me.
edit: Meant, even if you lose you have cost them significantly an that has the intended effect of signaling they should improve customer service.
Very typical from Google.
Unfortunately Google as a company and their corporate practices are nothing like that.
But it's pretty easy to look a bit into the future, and predict what we would see during "Extend and Extinguish" phases of TensorFlow project.
Also AdSense, but there's obvious reasons they support that.
> They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options.
Sounds like Google did a great job with Firebase. Definitely gives you that Google-feel of unresponsive support and no hope when you have problems with any of their services. I bet they also have a "community" support forum or Google Group full of helpless people talking to a virtual wall. (Edit: Yep, they recommend Stack Overflow, Quora, and then their Google Group.. lol. Some things never change)