So essentially, you have a system where people spend other people's resources for living and their success is judged by making the chain link above happy. In especially large companies it's easy to have a disconnect from the product because people in the top specialise in topics that have nothing to do with the product. If the people at the top want to have this shiny new thing that the press and everyone else is saying that it's the next big thing, you better give them the new shiny thing if you want to have a smooth career. In publicly traded companies, this is even more prevalent because people who buy and sell the stocks would be even more disconnected from the product and tied to the buzzwords.
The more technical minded people who have the hunch on tech miss the point of the organisation that they are in and get very frustrated. It's probably the reason why startups can be much more fulfilling for deeply technical people.
This is one of the reasons I roll my eyes whenever I read something like "McKinsey says 75% of Big Data/AI/Buzzword projects do not deliver any value." What's the baseline for failing and/or delivering zero value because those projects were destined to fail?
The whole point is, from their point of view those decisions are rational. It's much more lucrative from their (managers') personal point of view to develop a smokes-and-mirrors looks-good-on-ppt AI project. To be safe from risk, don't give the AI people too much responsibility, let them "do stuff", who cares, the point is we can now say we are an AI-driven company on the brochures, and we have something to report up to upper management. When they ask "are we also doing this deep learning thing? It's important nowadays!" we say "Of course, we have a team working on it, here's a PPT!". An actual AI project would have much bigger risks and uncertainty. I as a manager may be blamed for messing up real company processes if we actually rely on the AI. If it's just there but doesn't actually do anything, it's a net win for me.
Note how this is not how things run when there are real goals that can be immediately improved through ML/AI and it shows up immediately on the bottom line, like ad and recommendation optimizations in Youtube or Netflix or core product value like at Tesla etc.
The bullshit powerpoint AI with frustrated and confused engineers happens in companies where the connection is less direct and everyone only has a nebulous idea of what they would even want out of the AI system (extract valuable business knowledge!).
The useful AI/ML isn't glamorous, it's quite boring and ugly. Things like spam detection, image labeling, event parsing, text classification.
It's hard to get a big, shiny model into direct user facing systems.
Either way I don't think it matters too much because people can't really tell simple from shiny as long as the buzzword bullet points are there.
The point is rather that the job of the data science team is to deliver prestige to the manager, not to deliver data science solutions to actual practical problems. It's enough if they work on toy data and show "promising results" and can have percentages, impressive serious charts and numbers on the powerpoint slides.
I've heard from many data scientists in such situations that they don't get any input on what they should actually do, so they make up their own questions and own tasks to model, which often has nothing to do with actual business value, but they toy around with their models, produce accuracy percentages and that's enough.
https://www.burning-glass.com/u-s-job-postings-increase-four...
Looks like all job postings "collapsed during the pandemic"
It's important to understand why stuff fails. That's the only way to stop things from failing in the future and make sure people are on the right path to not failing. If a large number project failures are management failures, it's useful to know that. Otherwise you try to fix everything except the management structure.
"Personalization" is in a similar place for digital publishing; everyone wants it, products and services carry big price tags, and few organizations want to invest in foundational work or simple, iterative improvements. So they swing for the stars with unrealistic goals like "micro-targeted messaging perfectly tailored to every visitor, no matter where they are in the customer journey" and the results are predictable…
I take the increasingly grim accounts of project failure rates from analyst firms as a good sign — they can be used to sober up executives with unrealistic dreams.
These claims are usually high level and based on surveys or whatever. Failing usually means leadership gave up. As far as high level awareness of project success rates, it's probably accurate enough to justify the point: companies are generally bad at doing X. This tends to be true for many different kinds of X, because business is hard.
I generally don't agree that people make up destined to fail projects for selfish gains. I'm sure it happens, but that seems bottom of the barrel in terms of problems to fix. With DS specifically, leaders just don't know what to do. So they hire data scientists, and the data scientists don't know anything about the business, so they make some dashboards or whatever and nobody uses them. It's really not easy. Business is hard.
Similarly, you couldn’t just fake these types of savings because they needed to be showing up in budget requests. If I saved $10M in hardware costs, then that line item in the budget better reflect it.
AKA the principal-agent problem:
https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...
https://en.wikipedia.org/wiki/Putt%27s_Law_and_the_Successfu...
https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_ir...
Source: https://www.interviewquery.com/blog-do-they-want-a-data-scie...
I think the opposite is just as often true: Startups often don't have any real customers, so it's all about buzzwords and whatever razzle-dazzle they can put in a pitch deck to raise the next round.
The place I currently work is data-driven (perhaps to a fault). Every change is wrapped behind an experiment and analyzed. Engineers play a major role in this process (responsible for analysis of simple experiments), whereas the data org owns more thorough, long-term analysis. This means there are a significant number of people invested in making numbers go up. It also means we're very good at finding local maxima, but struggle greatly shipping larger changes that land somewhere else on the graph.
Some of the best advice I've heard related to this is for leadership to be honest about the "why". Sometimes we just want to ship a redesign to eventually find a new maximum, even through we know it will hurt metrics for a while.
...probably true to some extent, but not all leaders are self important ass hats who refuse to acknowledge they are simply “making decisions” not “making good decisions”. Most leaders are doing the best they can (often even very well) with the insights available to them.
I don’t think most data teams are really at fault; they’re just doing what they’re told.
The problem imo lies with the analysts who fail to do anything useful with the data they’re given, and demand constant changes from the data team because they want to deliver silver bullet results to the leadership level.
That’s the problem layer; people who want to be important but have nothing to offer, whipping their data team to produce rubbish and then blaming them for either a) not producing anything fast enough or b) not making the numbers big enough.
"Data science"/analytics groups are a cost center telling management things they do not want to hear, and disrupting management narrative (with receipts). There's no point to it; either you deliver "moneyball"-like opportunities that are ignored, or torture the data to fit existing narratives. In both cases, you eventually get stabbed by an experienced bureaucratic knife fighter.
So, I switched from predictive analytics and put on my prescriptive analytics hat. Over the time I was there I created several presentations containing multiple paths forward letting management feel like they were deciding the path forward.
This continued until I was fired. The board didn't like that I wasn't using a neural nets to solve the companies problems. Startups often do not have enough labeled data, so DNNs were not considered. Oddly, I didn't get a warning or a request about this before being let go. I suspect management got tired of me managing upward. In response my coworker quit right then and there and took me out to lunch. ^_^
Then the company doesn't listen to either group of people (neither tier 1 sales/support people, nor the ML people) and then fires / shuts down the entire division because "upper management didn't find value"
A lot of time they are right to ignore it. I've seen tables say X, but there was some flaw up the capture stack. Very few data analyst have the broad based knowledge and dedication needed to trace the data stack to establish the needed trust with the executive team.
I would have expected the researchy people to be better at it, as often you'll need to collect and analyse your own data during grad programs, and thus have some experience.
If you give executives data and they don't like the results, they will ask you to tweak parameters until the data represents what they want.
And there‘s the point where - IMHO - 3% gain may not be profitable enough.
Here's another aspect - in many places nobody listens to the actual people doing the work. In my last job I was hired to lead a Data Science team and to help the company get value of Stats/ML/AI/DL/Buzzword. And I (and my team) were promptly overridden on every decision of what projects an expectations were realistic and what were not. I left, as did everybody else that reported to me, and we were replaced by people who would make really good BS slides that showed what upper management wanted to see. A year after that the whole initiative was cancelled.
Back in 2000 I was in a similar position with a small company jumping on the internet as their next business model. Lots of nonsense and one horrible web based business later, the company failed.
It's the same story over and over again. Some winners, lot of losers, many by self-inflicted wounds.