The whole codebase is riddled with the same kind of layering but we do now have guidance about doing stuff like that ("DON'T") and a plan to simplify some of the worst offenders (like the multi-layer `Validate` hell hole.)
I don't say what you describe doesn't happen, but it's my impression that most people try to adopt solutions that minimize costs and development time (which also translates to money). 99% of the time it's not "do the best thing to satisfy solve this problem" but it's "solve this problem as fast as possible, without adding additional costs and using as few developers as possible".
Agree with that but from my experience that's more like 20% of the time. The rest is the various kinds of bullshit development where people are padding resumes, having boss's pet hobby horse forced on them, latest shiny flimflam, etc.
(A decent chunk of that 30 years has been contracting and that tends to be at places with problems which might be biasing my sample set.)
Or people in this industry are geeks and curious and like to try new stuff and technologies just for fun?
That's the case for the majority of people I've worked with.
Sure but do that at home or on POCs. Not on production code.
There's also "blinker-driven development" where people pick the solution based on their own personal set of hammers rather than, again, something more suitable.
(There's loads of these though - e.g. "optimisation-driven development" where the solution MUST GO BRRRRR even if the problem could be fixed by Frank typing in "Yes" once a week. "GOF-driven development" where everything has to rigidly fit into a GOF pattern regardless of whether it actually does. "Go-driven development" where everything has to be a interface and you end up reading a method called Validate which calls an interface method Validate which calls an interface method Validate which calls an interface method Validate and you wake up screaming every morning because why just wtf why please help me pleasehelp)