E.g., >90% of engineers would agree enterprise-fizz-buzz is over-engineered.
But perhaps ~50% of engineers agree my SaaS app is over-engineered.
It is different in the physical world where materials are a big factor and engineering will definitely not make free use of them. There are inefficiencies here and there but they're largely reduced. In the digital world of programming accidental and intentional complexity often slips through without anybody noticing and that is for many reasons. It's a relatively new field and is still quite inefficient. I worked in many places whose codebases are a giant maze designed with no clear architecture and sometimes I suspect this complicated mess of intentional moat building engineering.
What an engineer is actually there for is to run the numbers and predict behavior without having to go through the mess or expense of doing it any more than absolutely necessary. To plum the depths of our collective technical knowhow to drive a project that makes all the right tradeoffs.
It's not about making a bridge that can hold a semi out of matchsticks and bubblegum. It's about knowing it won't work without having to do it to find out.
Spending time and money looking for lighter materials even when the bridge is designed to support a lot more weight than traditional time tested materials. Justification : "Maybe cars will get heavier, and bigger saving 3 pounds is a good pursuit"
I don't think over-engineering is the only problem: shiny-new. Shiny-new syndrome wherein the typically junior programmer cannot help but jump from one new thing to the next, advocating that some new system must use the new tech that they recently discovered. While the enthusiasm is ok, working with people like this can be a drag.
Another problem: we are facebook too! In the sense that small companies heartily believe that they actually have problems at the scale of a company like facebook (or will ever get to that level). Again, hearing someone say, "We should do xyz because Netflix does it!" makes me want to give up on this profession.
Precisely! It's tiresome how many people setup Kubernetes for a network of 2-6 computers.
The only problem it has (and why I don't use stuff like kubernetes at home) is that the configuration formats aren't stable and thus require a lot of ongoing maintenance.
A coarse rule of thumb from a few decades of my own experience in software development is that version 1 will be quick, dirty and (in retrospect) naive. Version 2 will improve it with what you learned the first time around. And version 3 tends to be the clean, solid, polished winner. I've learned to expect and encourage refactoring to arrive at this point.
In my experience, you often start with a somewhat over engineered solution. Then you take the (long) process of really understanding it and slowly distill it down. This is true for a new feature and also the whole codebase.
There are other coping mechanisms you can use, although some of them are arguably worse than procrastination (eg overengineering).
This is based on the assumption that more thinking == better understanding.
We have this one technical coding challenge. Most of the submissions are 1000+ LOC and quite heavily engineered. One submission though was 300 LOC, runs 3x as quick as everyone elses and is the only one to get 100% in our acceptance tests. The author was very self-deprecating about it - describing it as a quickly cobbled together submission.
I'm nearly 40 and find I over analyze the design of everything. Which is great when I'm architecting a high level software feature, but when I get to coding I'm almost at analysis paralysis over every, damn detail. I miss that sense of flow.
Now you probably shouldn‘t code golf either, but KISS works as an operating principle for a reason.
In the last few years, I started shifting my focus from what variety of cases to handle with an elaborate solution, to what possible cases I might be blocking with a simple solution. Gratuitous YAGNI was an important intermediate step in that process. It might not be very FAANG-friendly (or it might not scale to four-digits-engineers regardless of brand), but both in terms of real life performance and peer feedback, it's yielded very good results.
While the first is prone to becoming a bottomless pit of SWE-self-pleasuring; the latter is actually very useful in coming up with something that's cheap, easy and maintainable, but still flexible enough -- which doesn't necessarily mean extensible! Ease-of-refactoring is another flavor of flexibility, and that's where I found this kind of thinking beating YAGNI alone.
One actual drawback I see is that it makes DRY difficult to achieve, but the more people I work with, the more I'm becoming disillusioned with that anyway. It's such an overly adaptable dogma to generate work that is often pointless, and becomes harmful so easily. I think we found so much comfort in "don't type the same thing twice" (though I still try to keep it to 3) that we don't stop to think about "don't maintain the same information twice" anymore. But this is 3AM rambling, so I'll stop here.
You've already seen how to be a better SWE than most.
Now you just have to execute.
Many interview processes seem to favor how well a candidate can enumerate edge cases and problem spaces over effective risk assessment and cost management. They're both important to evaluate but often in practice the dumb solution is what my team ends up using because can be more maintainable, cheaper to build, easier to reason about, etc. Today my aim is to get my requirements, write as few lines of quality code in as short of a time as possible, test it, ship it and be done.
Narrow focus and the ability to scope things down to what exactly what matters helps a lot. I defeat over-analysis by meditation, intentional dumbness/willful ignorance, and flow state.