What about thousands of users that will not see this reply? The gymnastics that an average user needs to do in order to contact support are ridiculous, and according to many stories here, it mostly solves nothing. Your explanation is fair enough, but I assume you guys should make a solid plan for increasing support quality and make it more available.

As of wrongly billed resources - to be honest, you are half blamed for that. I often create apps and base on SAAS reporting to see if the script behaves as expected and uses as many resources as needed. You have misled developers to believe everything is alright and now you are punishing them. I would expect some credits for time being and "transition" period.

All fair points and things we're working on.

We do often give credits to help in situations like this, though I can't speak to specifics publicly. We're working with this developer to try to resolve the issue, and we'll do so with other users who run into similar problems. This user did contact support, but we dropped the ball here. We'll be reviewing our support to find other devs we might have missed.

Your responses are encouraging. Hopefully the customer service example you set today will be followed by the rest of Google!

I'm sure it will be followed the next time a post reaches the front of HN

This problem was posted on stackoverflow 9(!) months ago http://stackoverflow.com/questions/38959321/firebase-databas... It has been updated recently: "After 9 months of back-and-forth emails (over 40 emails), Google has acknowledged that they have found some bugs that may be responsible for high bandwidth usage, but bandwidth usage is still too high. Resolving this issue does no appear to be a priority for Google/Firebase (it took them 1.5 months to respond to the last email)."

Andrew, Firebase is great, you did a great job and looks like you are continuing to do so by engaging directly like now. Having said that, it's time you, your team or even the entire google start looking seriously at automating your support workflow. How on earth can such a popular product ignore support requests, whether it's via email or any other medium?, especially for paying users. You should have a CRM/CSM tool that should automatically assign tickets based on the email received and if not resolved within a time period should escalate directly to senior person(s) until it's acted upon.

I can only guess from my limited experience talking to people at Google and reading about their systems, but I would guess the problem is over-automation of customer support.

I also wonder who shifts through the tickets themselves. Is it the engineer straight out of Stanford with 150k combined compensation and no serious training in customer satisfaction? The curious developer who built the product and wanted to see what people thought of his/her baby? A lower paid employee in a country with a lower cost of living? Some combination of the three? I feel like the culture at Google might preclude it from ever having good customer support except for the highest paying customers.

Clearly we have some work to do here!

This has been the case with every google product

I think you meant Google project.

Googles real products are made out of meat [1]

1. https://m.youtube.com/watch?v=7tScAyNaRdQ

I don't understand.

> The Firebase Realtime Database charges for SSL overhead on all requests ... We’ve always had a policy of charging this way.

So it's always been charged.

> Unfortunately, we introduced a bug late last year that began undercharging for SSL

Except it hasn't always been charged?

> We recently started actually enforcing overages on legacy plans and our current fixed-price ($25/mo Flame) plan.

In fact, if you're on a Spark or Flame plan, it's never been charged?

I feel like this response is straight from Office Space or something. "So we just went ahead and fixed the glitch."

My interpretation: 1) they always charged for SSL overhead; 2) they introduced a bug in late 2016 that underestimated SSL overhead; 3) that reduced charges for SSL overhead, and also broke flagging of Spark and Flame plans with too much SSL overhead; 4) they just fixed the bug, restoring normal estimates of SSL overhead; and 5) that restored accurate charges for SSL overhead, and also restored flagging of Spark and Flame plans with too much SSL overhead.

I'm guessing that OP's SSL overhead spiked after the measurement bug was introduced. Maybe also Firebase wasn't enforcing their limits before introducing the measurement bug, because they weren't confident about the data.

For a service that is billed based on a metric that only be measured by the provider, that provider has an important responsibility to accurately measure the metric, because important financial and technical decisions will be made based on that information. The fact that the metric was under-estimated (and therefore the costs were lower than they should have been) certainly constitutes some kind of compensation, the actual damages might be far beyond the temporary reduction in costs.

A feature might be rolled out "as is" to the entire user base after it was determined, on a small sample, that the costs per user were acceptable ; if the costs were greatly under-estimated, the company is left with a choice of discontinuing the feature (angering users) or scrambling to optimize the software, possibly starting from scratch with another technology. Or a yearly budget might have been designed based on usage estimates, only to be revealed as insufficient once actual estimates become available, and requiring an emergency cash injection for the company to pay for it.

I'm not a user of Firebase, but if our cloud provider (Microsoft Azure) were to announce that our costs had been underestimated by 2x or more, we would sue and argue that a 50% cost reduction for the past few years does not even begin to cover the strategic impact of the incorrect metric.

It's great that you're going to help the one guy who amassed a bully pulpit here on HN.

I have a question though...Why does Firebase charge ONE DOLLAR per gigabyte? 1995 called and wants their business model back.

This alone has limited my use of Firebase to Dynamic Linking for my app. Not that my app is going to smash any records, but paying customers are a good thing to get, aren't they?

The $1/gb includes all queries, something that you'd get charged for with say DynamoDB for both read/write queries and for reserving capacity.

That said I would be curious why storage is so expensive compared to DynamoDB ($5/gb/mo storage vs. $0.25/gb/mo).

It's at least potentially valid to suggest that charging the same rate for SSL overhead as for normal traffic is maybe not the best system, though, since I'm assuming that the overhead does not affect load on the value-adding servers but rather at a much more straightforward piece of the networking stack (LB or wherever SSL terminates before data gets into the proper Firebase stack).

Say, the $0.12/GB that Google Cloud Functions uses, since there's no value to the user or cost to the provider beyond what you'd see there (and indeed, one solution seems to be to simply route through cloud functions).

Is that possible, mayop100, or am I misunderstanding the system? Obviously it's not up to your users to dictate your pricing, of course, but if it's feasible it would probably earn a lot of goodwill.

> We’re embarrassed by the level of communication on our side, and we’ll be working directly with this developer to resolve the issue.

That gets to the heart of the issue -- I think most people believe that if there had been an effective channel for communicating then the issue would never have been such a problem. But "we'll be working directly with this developer" only solves ONE PERSON'S problem. The message I would like to see you taking from this is that your team needs a better way for ALL users to reach out for support in those situations where the support is necessary.

What support? Google has no support, ever. You care about "users" but don't give a shit about any one user.

I'm starting to really fucking hate Google.

Google doesn't have support for free products but they certainly do for paid. Whether the support is good or not is another question but it's there.

Just go to https://support.google.com/googleplay/?hl=en and click "Contact Us" in the top right. All the menu items I clicked that don't have straightforward solutions went straight to a menu where within 2-3 minutes I can get live chat or a phone call.

I've also had no problems getting help with my paid Google Apps account when I need it. For cloud there's paid support with a 4-hour response time: https://cloud.google.com/support/

We pay $150 a month for google's "silver" level support for GCE, appengine etc.

It's consistently very good - they helped us with code debugging, performance optimization, load balancing and other fairly complex issues that typically need an experienced engineer.

They do have support. I was stuck once with a friendly fella, that spent 45+ minutes talking about weather and weekend adventures to me, rather than solving my issue. It took less than 5 minutes to solve the issue, almost an hour of talk time. I would even bet money that the guy was heavily stoned.

At the end he asked me to provide positive feedback to him in exchange for friendly service. He said he will redirect me somewhere and I should give him a proper feedback. This is even while I was telling me I have no time and this is an urgent matter. At the end of the call it turned out it took him 5 minutes to solve the issue and he just wanted to chat while I was stuck at talking to him, thinking he is waiting for a solution to propagate.

I can't say this is a good support.

This is what happens when you try to drive work-morale with a stick and carrot approach. Teach your workers good morale and lead with good example, every customer is important. But also your employees.

Google's customer support for Project Fi has been absolutely fantastic from my perspective, and is very flexible.

It is hard to believe that tools to prevent overage use in the $25 plan haven't been developed earlier. All the necessary components were there: It has always been possible to track the use of resources per user, and it has at least in the early version of Firebase been able to send out warning emails if the enforced limits are approaching. So, to me it seems like a no-brainer to combine these two, in particular for a large company like Google.

This feels like a way to make more money off of developers. If you can't set a limit, you might end up spending more.

Andrew, you are a credit to Firebase, and I salute you for dropping in here and apologizing for the situation. Though you'll get some heat, I think your attitude of humility is a rare one these days, and much appreciated.

At the expense of revenue Google could lower the price to before the SSL billing fix to keep developer mindshare. There's at least one viable startup whose whole basis was baked on the 25 bucks a month use case.

just out of curiosity, is there a way to set a spending limit? Or at least an alert when crossing a spending threshold?

So the sudden 7000% cost increase is just the normal course of business due to a bug? Wow, great business plan -- tell them the first one's free ..then they pay.