News for hackers. Stuff that matters.
Re:How we spent $30k in Firebase in less than 72 hours by cloakandswagger on Wednesday August 01, 2018 @10:09PM (#17668758)
Basic metrics around platform API calls would have done the trick, no need for fancy container solutions.

edit: I would have also assumed they would get this for free from GCP's billing breakdown, but I'm not familiar with it. My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.

Re:How we spent $30k in Firebase in less than 72 hours by dkoston on Wednesday August 01, 2018 @10:18PM (#17668803)
Exactly. Not having proper tooling meant they didn’t know what to do. When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code.

Google cloud has trace built in which could have shown them execution times and is dead simple to drop into most frameworks.

The real story here is that hey didn’t have engineering leadership on the team who knew how to properly diagnose issues, put tooling in place before launch, and understand how their system is architected.

Kudos to the engineers for solving this issue under pressure.

Re:How we spent $30k in Firebase in less than 72 hours by chrisweekly on Thursday August 02, 2018 @02:09AM (#17669540)
>"When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code."

^ Best response, hands down.

Re:How we spent $30k in Firebase in less than 72 hours by themarkn on Thursday August 02, 2018 @07:19AM (#17670623)
> My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.

They didn’t upgrade packages to solve the mystery billing. They upgraded packages before they checked what was going on with the database. When they saw the high billing that pointed them to the problem and they fixed it.

There was some questionable judgement shown by not checking db requests first, sure, but in no way did someone think “our google billing is high, we better upgrade angular”.