The last 7 years of my life as an SRE. It's maddening.
It was the pet project of another team-lead, that no-one else wanted. Group decision making is bizarre.
Cost can be a difficult problem to solve! I've been working on a project for about six months trying to identify just how much it costs to deliver a unit of the thing we sell. There's just so much variability, one customer might have widgets that are 40kb in size and rarely ever run widget analytics workloads, while another customer has 40mb widgets and can't stop looking at them.
I feel for the whole "how do customers use the application" thing too. Product analytics at most places seems to be a concern long after features are developed.
"So how many customers have been using that feature we spent two years and $20MM developing?"
"Good question."
"Soooo, mywittyname, can you figure out how many people at companies with over 200 seats look at dog pictures after opening an email with a 'C' in the title?"
I think what saved us was the unlimited license we had negotiated for the database server we were using (which wasn’t bad technology), expired and then the company was going to charge us through the nose for any new servers.
That triggered a rearchitecture of our system, into micro services that made sense for the kinds of queries, volume, and load we were dealing with. We did have a lot of data, but many of the queries could be satisfied by a key value store, for example, which didn’t require the same amount of hardware resources.
The original project did lead us to products our customers wanted to buy. Then we just had to change the architecture to fit those actual products.
At the last company I worked at (won't call them a startup, just a 10-year old small business that somehow kept raising funding but has never made a profit) I was astonished to learn from a new CPO that in our tenth year we didn't know if we were making money from a customer.
We had no idea how individual customers used the application, which was very data intensive, or what that was costing us. The information was there, but no-one ever looked. Every sale was considered a success, even if it turned out to cost us 3x in data costs and customer support. Often customer support alone would make a sale into a loss.
We spent a relatively large portion of our revenue running very expensive servers to respond instantly to searches that no-one ever performed. I created detailed monitoring to show this. I talked about them to everyone, including Product and the CEO. No-one disputed those facts. But instead we had several people essentially dedicated to downsizing application servers and deleting unused S3 buckets, trimming three zeros a month when we could have trimmed five.
I have learned not to assume that warnings are visible, that people pay attention to them if they are visible, or that anyone cares.
They're still circling the drain, still raising money (somehow!) and I doubt anything's changed.