Basic metrics around platform API calls would have done the trick, no need for fancy container solutions.

edit: I would have also assumed they would get this for free from GCP's billing breakdown, but I'm not familiar with it. My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.

Exactly. Not having proper tooling meant they didn’t know what to do. When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code.

Google cloud has trace built in which could have shown them execution times and is dead simple to drop into most frameworks.

The real story here is that hey didn’t have engineering leadership on the team who knew how to properly diagnose issues, put tooling in place before launch, and understand how their system is architected.

Kudos to the engineers for solving this issue under pressure.

>"When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code."

^ Best response, hands down.

> My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.

They didn’t upgrade packages to solve the mystery billing. They upgraded packages before they checked what was going on with the database. When they saw the high billing that pointed them to the problem and they fixed it.

There was some questionable judgement shown by not checking db requests first, sure, but in no way did someone think “our google billing is high, we better upgrade angular”.

> modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails.

Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration.

It usually isn't hard to identify which component of your product is misbehaving. Before getting into complex magic of containers and sandboxes and stuff, I'd start with the easiest - looking at the Network tab, at your server's resource use, reading the logs, adding some log statements measuring times in suspected areas, actually profiling the backend code (e.g. with a statistical profiler). This should quickly help you identify where the problem is manifesting itself. Then the search for the cause begins.

There is also a common sense problem. Not trying to play the blame game but the guy who ran a subtotal displayed on the front page by having to download every row in the database probably isn’t a great programmer. It’s easy to say don’t write bugs, but there are bugs and moronic design decisions. This to me doesn’t feel like a real bug.

There's a good chance this is true, but this can happen even with competent developers. One person writes the unoptimized function figuring it can be improved later, someone else assumes this must be cheap and drops it on the front page. Or there was meant to be a caching layer, that didn't work because of some bug.

>Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers

Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything.

We've been understaffed for a year and we've been told not to hire any more staff until the new financial year.

I've got a "technical debt" item on the backlog but business drives the priorities and it'll never get done.

> Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything.

Only matters when the fault is located or manifests itself precisely in that layer, and that's always a risk. Consider all the third-party dependencies you use. They usually have only their APIs documented. Fixing a fault usually requires knowledge of the internal implementation.

> We've been understaffed for a year and we've been told not to hire any more staff until the new financial year.

> I've got a "technical debt" item on the backlog but business drives the priorities and it'll never get done.

Yeah, I get that. I've seen that. Thing is, you can only play lottery so long with your main product - and also, if your workplace runs an assembly line so tight that you can't spend hours thinking per ticket (excluding the most trivial ones), then something is seriously broken on yet another level.

Ultimately, I guess what I'm saying is that the main problem here is cultural - possibly both on developer and management side. The actual technical tasks aren't usually that challenging.

Well a browser is basically a sandbox for a webapp.

To debug something complex, I would use chrome devtools, which can measure all kinds of metrics, and the function "Audit > LightHouse" automates the process and ranks a webapp in several key categories.

This case would appear to be related to network requests, so that issue should be fairly obvious in LightHouse.