In the tech industry, our ability to obtain new tools and find new problems to solve vastly outstrips the speed at which we can properly understand either the tools or the problems. When relevance in an organization is tied to your ability to leverage the new sexy, of course the equibrium behavior is to be half-cocked 100% of the time.

I think it's a cargo cult problem. We see massive companies showing off their solutions to gigantic engineering problems that can only be solved in a certain way, and we think that we also must solve problems that way. Or that we'll get promoted if we solve problems "like <insert-big-name-tech-company>" in our architecture.

Chances are, all you need is to properly use a relational database or another similar, proven, highly flexible and optimized tools, and you'll be fine until you have the time and money to solve the scale issues.

I think it might go deeper than just the engineers, although we are definitely a large part of it. The basic issue is that most companies don't compensate employees for compentently building software.

Technical debt is always a future managers problem so it's both, never worked on and preventing it is never seen as a big accomplishment when evaluating an engineer.

Secondly managers are people and people like flashy things. When someone says they built the website in vanilla js because it works and we won't need something more complicated for years, the manager isn't going to get that excited or remember when raise time comes around. If the engineer says they rewrote an application from [legacy language] to some hip new language that the manager has heard about from their peers in other companies who are also rewriting their applications, it's going to stick in the managers head when they are rating their employees.

You get what you pay for and modern companies are paying for semi working but flashy software more than they pay for solid but boring software