How we spent $30k in Firebase in less than 72 hours

Re:How we spent $30k in Firebase in less than 72 hours by TeMPOraL on Thursday August 02, 2018 @01:30AM (#17669403)

> modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails.

Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration.

It usually isn't hard to identify which component of your product is misbehaving. Before getting into complex magic of containers and sandboxes and stuff, I'd start with the easiest - looking at the Network tab, at your server's resource use, reading the logs, adding some log statements measuring times in suspected areas, actually profiling the backend code (e.g. with a statistical profiler). This should quickly help you identify where the problem is manifesting itself. Then the search for the cause begins.

[ Parent | on HN ]

Re:How we spent $30k in Firebase in less than 72 hours by cm2187 on Thursday August 02, 2018 @03:33AM (#17669797)

There is also a common sense problem. Not trying to play the blame game but the guy who ran a subtotal displayed on the front page by having to download every row in the database probably isn’t a great programmer. It’s easy to say don’t write bugs, but there are bugs and moronic design decisions. This to me doesn’t feel like a real bug.

[ Parent | on HN ]

Re:How we spent $30k in Firebase in less than 72 hours by dmurray on Thursday August 02, 2018 @04:35AM (#17670051)

There's a good chance this is true, but this can happen even with competent developers. One person writes the unoptimized function figuring it can be improved later, someone else assumes this must be cheap and drops it on the front page. Or there was meant to be a caching layer, that didn't work because of some bug.

[ Parent | on HN ]

Re:How we spent $30k in Firebase in less than 72 hours by voltagex_ on Thursday August 02, 2018 @03:33AM (#17669802)

>Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers

Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything.

We've been understaffed for a year and we've been told not to hire any more staff until the new financial year.

I've got a "technical debt" item on the backlog but business drives the priorities and it'll never get done.

[ Parent | on HN ]

Re:How we spent $30k in Firebase in less than 72 hours by TeMPOraL on Thursday August 02, 2018 @07:37AM (#17670692)

> Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything.

Only matters when the fault is located or manifests itself precisely in that layer, and that's always a risk. Consider all the third-party dependencies you use. They usually have only their APIs documented. Fixing a fault usually requires knowledge of the internal implementation.

> We've been understaffed for a year and we've been told not to hire any more staff until the new financial year.

> I've got a "technical debt" item on the backlog but business drives the priorities and it'll never get done.

Yeah, I get that. I've seen that. Thing is, you can only play lottery so long with your main product - and also, if your workplace runs an assembly line so tight that you can't spend hours thinking per ticket (excluding the most trivial ones), then something is seriously broken on yet another level.

Ultimately, I guess what I'm saying is that the main problem here is cultural - possibly both on developer and management side. The actual technical tasks aren't usually that challenging.

[ Parent | on HN ]