SaaS companies pricing is often geared towards startups that "can't afford" to control costs, i.e. developer time spent on minimizing cost has a higher monetary and opportunity cost than ignoring those costs in the short term, even if they're piling up at $50/mo for every auxiliary service.
But that strategy does not account for those same startups after they grow enough to take a breath and start optimizing costs.
The “how” question has changed every 3 years in the last 10 that I’ve done data work. That trend is driven primarily by data scale and partially that analytics and ML access patterns have evolved a lot.
This post [0], and the open job requisitions suggest you’re using a bit of cash to do this. What is your business model?
0: https://mobile.twitter.com/grouparoo/status/1292844163617239...
Are you eventually running your own cluster? Or, are you paying per Gb per second how GCP bills with decoupled compute / memory / storage? Or maybe with a little bit more overhead with the orchestration cost on top of consumption with how azure bills for ADF.
Just from reading blog posts, Spotify’s move was interesting, regarding their annual “songs you liked” at the holidays, where they sort of ran a hybrid.
Sorry to digress but what is the end state. Is there just a number where it makes sense to hire people and build and/or shift vendors? And if yes, what vendor is “after” stich? Astronomer maybe. Sorry if that’s basic.