This is one of the reasons I roll my eyes whenever I read something like "McKinsey says 75% of Big Data/AI/Buzzword projects do not deliver any value." What's the baseline for failing and/or delivering zero value because those projects were destined to fail?
The whole point is, from their point of view those decisions are rational. It's much more lucrative from their (managers') personal point of view to develop a smokes-and-mirrors looks-good-on-ppt AI project. To be safe from risk, don't give the AI people too much responsibility, let them "do stuff", who cares, the point is we can now say we are an AI-driven company on the brochures, and we have something to report up to upper management. When they ask "are we also doing this deep learning thing? It's important nowadays!" we say "Of course, we have a team working on it, here's a PPT!". An actual AI project would have much bigger risks and uncertainty. I as a manager may be blamed for messing up real company processes if we actually rely on the AI. If it's just there but doesn't actually do anything, it's a net win for me.
Note how this is not how things run when there are real goals that can be immediately improved through ML/AI and it shows up immediately on the bottom line, like ad and recommendation optimizations in Youtube or Netflix or core product value like at Tesla etc.
The bullshit powerpoint AI with frustrated and confused engineers happens in companies where the connection is less direct and everyone only has a nebulous idea of what they would even want out of the AI system (extract valuable business knowledge!).
The useful AI/ML isn't glamorous, it's quite boring and ugly. Things like spam detection, image labeling, event parsing, text classification.
It's hard to get a big, shiny model into direct user facing systems.
Either way I don't think it matters too much because people can't really tell simple from shiny as long as the buzzword bullet points are there.
The point is rather that the job of the data science team is to deliver prestige to the manager, not to deliver data science solutions to actual practical problems. It's enough if they work on toy data and show "promising results" and can have percentages, impressive serious charts and numbers on the powerpoint slides.
I've heard from many data scientists in such situations that they don't get any input on what they should actually do, so they make up their own questions and own tasks to model, which often has nothing to do with actual business value, but they toy around with their models, produce accuracy percentages and that's enough.
https://www.burning-glass.com/u-s-job-postings-increase-four...
Looks like all job postings "collapsed during the pandemic"
It's important to understand why stuff fails. That's the only way to stop things from failing in the future and make sure people are on the right path to not failing. If a large number project failures are management failures, it's useful to know that. Otherwise you try to fix everything except the management structure.
"Personalization" is in a similar place for digital publishing; everyone wants it, products and services carry big price tags, and few organizations want to invest in foundational work or simple, iterative improvements. So they swing for the stars with unrealistic goals like "micro-targeted messaging perfectly tailored to every visitor, no matter where they are in the customer journey" and the results are predictable…
I take the increasingly grim accounts of project failure rates from analyst firms as a good sign — they can be used to sober up executives with unrealistic dreams.
These claims are usually high level and based on surveys or whatever. Failing usually means leadership gave up. As far as high level awareness of project success rates, it's probably accurate enough to justify the point: companies are generally bad at doing X. This tends to be true for many different kinds of X, because business is hard.
I generally don't agree that people make up destined to fail projects for selfish gains. I'm sure it happens, but that seems bottom of the barrel in terms of problems to fix. With DS specifically, leaders just don't know what to do. So they hire data scientists, and the data scientists don't know anything about the business, so they make some dashboards or whatever and nobody uses them. It's really not easy. Business is hard.
Similarly, you couldn’t just fake these types of savings because they needed to be showing up in budget requests. If I saved $10M in hardware costs, then that line item in the budget better reflect it.
AKA the principal-agent problem:
https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...
https://en.wikipedia.org/wiki/Putt%27s_Law_and_the_Successfu...
https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_ir...
Source: https://www.interviewquery.com/blog-do-they-want-a-data-scie...
I think the opposite is just as often true: Startups often don't have any real customers, so it's all about buzzwords and whatever razzle-dazzle they can put in a pitch deck to raise the next round.
So essentially, you have a system where people spend other people's resources for living and their success is judged by making the chain link above happy. In especially large companies it's easy to have a disconnect from the product because people in the top specialise in topics that have nothing to do with the product. If the people at the top want to have this shiny new thing that the press and everyone else is saying that it's the next big thing, you better give them the new shiny thing if you want to have a smooth career. In publicly traded companies, this is even more prevalent because people who buy and sell the stocks would be even more disconnected from the product and tied to the buzzwords.
The more technical minded people who have the hunch on tech miss the point of the organisation that they are in and get very frustrated. It's probably the reason why startups can be much more fulfilling for deeply technical people.