That’s insane, how do the people in the org justify their work for promotion? There isn’t any need to map it back to observed top line metric shifts even for a 30 person effort?

“Person xyz built a complex model deployment system, capable of updating the recommendation engine in real time, with automated rollbacks and A/B testing. He leveraged 4 junior engineers and collaborated with multiple data science teams”

Indeed, this is why corporate KPIs exist.

Requiring everything to move KPIs is psychotic, but if your effort should have moved a KPI, and didn't, then you'd better have a damn good story about how it delivered value elsewhere instead.

Did people working on that engine know that their work is being ignored?

Yes…but the thing is downstream systems are dynamic and real time, so “could” use the recommendations, but never actually do. Hypothetically…

This type of thing goes on all over the company. Another example, a complex “data provider platform”. Which is actually just spinning up spark clusters, joining data, and making it available to downstream consumers. There’s a lot of bells and whistles, notifications built on top. Strip away the details, it’s just doing a join. Ten engineers, two senior engineers making 500k, long hours.

That’s Amazon.

I encountered the same sort of problems at amazon, I felt like it was because leadership didn't incentivize those optimizations to be made across the org. We'd identify some team that was the only one that could help us with something and nobody would lift a finger for us. Teams who were on the hook would ship us incomplete and untested APIs claiming they were done in order to work on other stuff. Managers didn't want to bring it up with other teams because of political ramifications.

I think you are describing all big tech companies. But what's the solution? I think it's just a result of having a large org. 20% of the org delivers 80% of the value. But when risk appetite is high and the company can afford to throw money at experiments, it makes sense to still pay for the rest of the 80%. When times are tough, it doesn't.

Definitely it makes sense to take bets on new tech. But a lot of higher up managers are not running these projects in good faith. At some level they know the projects are futile, and many around them are playing a game acting otherwise

How do you know that? I don't disagree that many projects are futile. But it is usually only apparent in hindsight, and humans are very good at justifying their own jobs and by extension, their team's jobs.

> But what's the solution? I think it's just a result of having a large org.

Right. Maybe Elon's approach is correct, and all these layoffs are a chance for companies to trim the fat and focus on what matters.

We're only now realizing the bubble the tech industry was in, and that pursuing endless growth is not sustainable.

> But when risk appetite is high and the company can afford to throw money at experiments, it makes sense to still pay for the rest of the 80%.

Does it, though? You're paying for full-time engineers to work on bets that might be completely misaligned even if they succeed, which also might be unlikely. And this makes up the majority of your company? It sounds ludicrous in any market.

How does a team of 30 not measure the usage of their system?

req/sec can be still high, it's just that the downstream service decides not to use the data

When the data your team serves is copied from service to service to provide high availability and no centralized failure point.