In most non-tech companies, it comes down to the motive of the manager and in most cases it is expansion of reporting line and grabbing as much budget as possible. Using "simple" solutions runs counter to this central motivation.
This is also true of tech companies. Witness how the "GenAI" hammer is being used right now at MS, Google, Meta, etc.
- the manager wants expansion
- the developers want to get experience in a fancy stack to build up their resume
Everyone benefits from the collective hallucination
That's the tech sector in a nutshell. Very few innovations actually matter to non-tech companies. Most companies could survive on Windows 98 software.
The flipside of this is that people at some places are fully aware of this and get very suspicious of consultants offering to handle their 'big data' challenges with loads of highly proprietary stuff.
Data organisation can be more of an issue, but the general issue with this is often a lack of internal discipline on the data owners to carefully manage their data. But that isn't nearly as attractive for management. Putting 'brought in new cloud vendor / technology' looks better than 'improved data organisation' on a CV even if the new vendor was a waste of money.
Assuming that it’s not me that’s being an ass (saying “I could do it with a few Python off a USB drive” would be being an ass), if my interviewer is offended because I offered the practical solution, I am not working there, especially if they’re from the team I’m going to work on.
It’s like a reverse signal.
I think it’s part of the BS cycle that’s prevalent in companies. You can’t admit that you are doing something simple.