1. I like interesting puzzles. A lot of code - especially commercial code - is pretty boring if you do it right. I find myself subconsciously pushing for features that will be fun to implement. And by "fun", I mean, features that will overcomplicate everything.
2. While I'm in the middle of programming something, all the choices that I make seem straightforward and necessary. Its only later when I step away from my code and then try to understand it with fresh eyes do I notice what a mess I've made of everything.
I also think that a lot of the time the most obvious solution to most problems is quite complex. It takes wisdom, skill and domain knowledge to know where to look for simple solutions to any given problem. Simple, clean solutions are rarely obvious.
"Oh, it looks like we're slowly implementing a halfbaked message queue. Lets use a standard message queue instead." "Oh, we're slowly building up a custom, buggy, binary framing protocol. Lets just use protobuf/msgpack". "How about instead of writing a custom RPC protocol to fetch data, we just use REST over HTTP. And then we can put nginx in the middle to cache our backend responses, and we can throw out our custom cache."
Developers grow. It starts with simple code that doesn't work. The next step is, complicated code that solves the problem, in messy unmaintainable ways. The next step is writing super clean, almost boring code, that's highly readable and "dumb" and does exactly what it's supposed to do.
The other thing to realize, a lot of great code doesn't just spring from the developer's hands in its final form -- it's extensively edited and rewritten into its final, good, form.
Not entirely facetiously, I think that, for engineers, there's real skill in creating simple solutions to simple problems—not, for example, finding the general instance of the problem and solving that, when the problem is unlikely to recur and crafting the perfect general solution delays delivery on what's actually in front of you.
(I know Perl's not fashionable any more, but I've always liked its design philosophy of "make easy things easy, and hard things possible." It seems like a slogan that can be adaptable to how to solve problems, though I'm not sure of the absolutely perfect analogue. Hmm, maybe I'm trying to solve the general instance of a problem ….)
This is true, but I think also illustrates the phenomenon. It's tempting not to solve the specific problem in front of you, because a more general problem might be easier and more elegant to solve—and this mindset can easily lead one into, at worst, never solving the problem; or, at less worst, solving a problem that's so far generalized that no one else looking at your code can tell why it's doing what it's doing.
(It can also happen that the more general problem doesn't have a simpler solution. If I want to print a string that has a few hard-coded values in it, formatted a particular way, I could develop a formatting spec and write a formatting library to process it, which is surely the right solution to the general problem—but, if the specific problem is likely only going to arise once, then it may be both easier to understand and a better use of time just to put in the hard-coded values.)
1. linked list is a generic data structure with a relatively simple interface
2. an application with 1k configuration values is generic in that it can handle everything, but is in no way simple.
By using protobuf/msgpack you lose the ability to precisely control the layout and encoding of your data on the wire. Most applications don't care, but this results in your wire representation being defined by "whatever protobuf says".
Say I want to transmit an unsigned 16 bit integer with protobuf. How do I do that? The documentation doesn't include 16 bit integers as a datatype, so I'd probably have to wrap it in 32 bits and/or use some varint stuff. It would be simpler to just write a big endian 16 bit int though.
I wish there was a simpler alternative to protobuf that gives more control to end users and doesn't try to be smart. Until then, making your own binary protocol is not over-engineering.
Something like "use fixed length 8/16/32/64 bit signed/unsigned integers in big endian, length prefix can be 8/16/32 bits, bool is 1 byte (00 = false, 01 = true)" etc, without extra stuff like varints or bit packing, which a lot of current formats are doing.
In short, just use the most straightforward way of encoding while also using the least amount of data. Big endian for ints is very common, simple and relatively compact if you only use the bit width that you need.
Developing good instincts for this stuff takes a lifetime.
That's what factorio is for!
That is a very good point. If your code doesn't look boring, then you probably doing something wrong.
Honestly for me over-engineering is usually the result of the opposite. Thinking too much when writing and having preconceived ideas about what a codebase ought to look like.
It was Casey from Handmade Hero IIRC who called his style of programming "compression based", effectively just writing code and factoring out what belongs together incrementally. Abstracting things out as they repeat, not consciously by design. I've taken this up more and more as a way to program.
Some fellow devs seem to live creating big beastly complex abstract PatternFactoryClassBuilderGenerators for simple one off use cases which should be quite simple.
Having devs and PMs on board with adding estimations and spending the time actually doing that refactor on the second or third time you're following a pattern is the tricky bit. It pays dividends long term though as you maintain velocity.
In my experience these are rarely, if ever, the reasons for over engineering.
It took me a month to realise that the staff developers very much revelled in and protected their bad code and bizarre domain choices.
It was so far gone that there was no way to get rid of them and the product was just slowly dying and burning the remaining cash.
Then the mergers happened and they all got let go, only retaining the name for brand power and the entire stack was quietly moved over to another similar product which was rebranded.
Separately, there were absolutely very large consultancies that had a programming style/rules based on making their implementations difficult to read/modify, needing to call their COE back in to fix their code or add features - with it being very hard to modify. Talking entire codebase structured with ridiculous levels of abstraction and annoying code style. Bad integrations requiring their tooling to work and make sense of etc.
They target traditional orgs where the management just wants to get a project through and then bleed them over years.
I think personality can account for this without any reference to incentives, which come in to explain how this personality problem can be so common among successful engineers.
But, for contemporary software developers in general, I'd guess more often it's either resume-driven development (e.g., add the big complex new framework keyword to your resume) or not yet having enough experience to know what complexity is worthwhile for a situation.
I had a team once implement the MediatR pattern for a < 5k LoC codebase (I'm guessing at its size but it was made larger due to the overhead of the MediatR pattern).
When I asked them to remove it, it became a political fight that went to the VP because they were convinced that sort of flexibility was a good idea. Fast forward a year and they have a new technical leader and he thought I wanted that complexity until we had a conversation and when I mentioned I didn't like it he confessed to me he wanted to rip it all out.
People value the wrong thing too often.
Believing flexibility is needed still suggests a lack of understanding.
This leads to a certain kind of decoupling, although with a higher tolerance for coupling that can kept in check by static analysis.
Values are those things that help guide you in the face of imperfect information. Not having a crystal ball that can predict the future perfectly means you have imperfect information.
To prioritize flexibility means that one lacks the understanding required to even build a misguided confidence.
At which point your description of a lack of understanding applies, hence my comment.
So much of the grug-brained argumentation is same-side violence, is developers roasting developers. Assigning bad motives & declaring immoral bad people among us probably happens some, and various petty or criminal over engineering happens some too. But I usually think there are simpler Hanlon's Razor explanations that aren't even personal, are just the story of an org: an organizational body where so many constituent members of the body-whole have so little idea what others are doing or have done, and access to so few who can offer informed situational-appropriate wisdom for whatever dark end a squad finds itself tasked with.
In some ways, this has been the actual story of open source. We get to grow communities of practitioners with individual knowledge & experience that ports & can be compared & discussed with others, from outside our company. We get peership that companies can rarely afford or find after they grow mid-sized.
The Influx folks just talked about their 3.0, & replacing their custom built engine with Flight, DataFusion, Arrow, and Parquet. Maybe they might miss one box or two on neat nitty gritty optimizations (maybe this guy elsewhere in this thread, insisting diy'ing binary encodings because protobuf doesn't have 16-bit ints can find some deficiencies, https://news.ycombinator.com/item?id=38078133), but the nice to haves of ecosystem integration seems like an intangible, but one that lets little good things out there plug in & help out, and that seems invaluable. https://news.ycombinator.com/item?id=38013714 https://www.influxdata.com/blog/flight-datafusion-arrow-parq...
The top comment to me epitomizes the worst of grug brainedness. It's smug & certain of itself, a strong & divisive Hanlon's violation of high degree. That kind of blue-on-blue attitude to me is far worse than 99% of the people who end up over engineering. Few organizations have real defense, lack coherency, to help themselves away from complexity, or they have such tight constraints that they overprune & disaffect those who do care and have good ideas. These are organizational issues, and having an org that can somehow see itself & what's afoot is the huge challenge. Stop hating on devs, please.
I don't think this makes sense. For example currently I have a big inclination to think bad about developers working for Vercel/recommending NextJs because they are invariably the same ppl and do it because of monetary benefits. The intentional over engeering they are adding to make a profit is insane and evil. There is no way I can make this point without "hating" the people doing it.
"Bad" design can be fine if it's kept simple.
I was thinking of good code as something I had to invent, but now I feel like it's more like something I have to discover.
Hah, I just experienced this when I asked my brother for a simple comparison of about 7 criteria in a 3 column table, to include in a report.
He gave me 12 criteria not entirely from the data source I was looking for, and a cost calculator to boot.
On the other hand, over the course of a few weekends he coded up a beautiful baroque bastard of an excel macro that saved us thousands of man-hours over 5 years.
No one can, and no one is. All lines of code are over-engineered, just some more so. No amount of thought will get you to the perfect solution; it's an unapproachable asymptote. We're all creatures of time and there's a limit to how many cycles we can afford to spend iterating in design space to try and hit the right note. (Not to say we can't get better with practice, though.)
The biggest factor I've found in overengineering is the lack of a long term roadmap. If you need to build a feature X, and you engineer the bare minimum and need to put in the same amount of hours to do X+1, your management is going to be upset that you're taking too long to ship. You already had it 80% (in terms of feature complete) of the way there, why is that extra 20% as hard as the first 80%? So the engineers build up scar tissue. If you have to handle a ton of cases that you don't understand, why not build a CaseHandlerFactory so you can add the next feature faster?
A clear roadmap of "this is what we want in 1 year" will help solve over engineering. Otherwise engineers are incentivized to make their code as configurable, modifiable, and extendable, as possible, regardless of cost or business need. Not to mention all the additional time trying to figure out "canonical" data models that will "future-proof" the applications interfaces. If you have to iterate quickly (which is not as common as agile folks wish), you need to build up a raport with leadership to help them understand that speed comes with trade offs: faster might mean 'more work to change later', while slower to market today might mean faster iteration down the line. These discussions ARE valuable for leadership, as sometimes they need a quick win because of a Q3 earnings hole or contract, and sometimes they are willing to make longer term investments.
You move too quickly, they call your solutions hacky, you move too slowly, they call it overengineered. Everyone needs to be on the same page of what the change is trying to do: win short term, win long term, or somewhere in the middle.
I've never understood this, maybe I'm just a bad engineer. I mean, it sucks having to pivot, but all those configurable and extendable pieces take hours trying to get the design right. And you only end up using ~1% of them, and then something you didn't foresee happens as well. I always end up spending months of my life saving days doing the change-overs.
And that's before we get into the problems with onboarding a new engineer (or being the one onboarded) into the kind of hellscape that the overly configurable application turns into.
Can subscribe to this, 16 years career. Even a horizon of 3 months seems unattainable.
I call it a "management problem", but maybe that's because I'm not in management..
The problem isn't usually the code, it's the people in charge not giving the right instructions, shifting the goal post, allowing feature creep, or the deadly sins of rewrites, large refactors, unrealistic deadlines.
Over-engineering is just fine in the real world. The problem is when that causes cost and deadline overruns. Those can be controlled for, even if the complexity can't. But finding a manager who knows how to manage software teams effectively and simultaneously keep his bosses from digging their own graves is even rarer than an engineer who writes simple code.
EDIT: I ironically wrote way too long here. Grug say better:
> note, this good engineering advice but bad career advice: "yes" is magic word for more shiney rock and put in charge of large tribe of developer
People can write perfect, simple, DRY code if they have the time to and are incentivized to. In most cases you're rewarded for launching the thing and showing one's "technical prowess" with the amount of work / intellect / design skillz™ required to launch the thing. The natural conclusion of this is that everything becomes a bloated, over-engineered mess of kludge solutions that gets rewritten every 3-7 years.
I haven't seen any data on this but I'd guess the "rewrite half life" is correlated with turnover / average tenure, so even if people tried harder to write not-over-engineered code, it'd probably get rewritten anyway. As a perfectionist, this truly bothers me, but I find sometimes thinking harder / spending more time on the best _simple_ solution is rewarded less than building the complicated overengineered thing. I'm sure better organizations exist but I have yet to find one in 9 years as a SWE...
Open Source projects are actually the best counter-example to this that I can think of, but even then the best libraries sometimes get rewrites or new versions when they've changed hands from one maintainer to another, and I'll note that financial incentives are very different between open source and the types of enterprise-y cruft most people working full time as SWEs on HN probably see. It's like comparing a well crafted academic paper to a lazily written work email.
In my experience, people trying to make code DRY also wind up writing over-complicated patterns and abstractions to make it so.
I think a large amount of over-engineering is likely due to people applying patterns where they don't need to, or building unnecessary abstractions, or otherwise doing what they think is "good code".
Most folks aren't incentivized or simply don't bother to think through things and try harder. The "this way looks more smart so I made it all complicated" thing definitely happens too, but in eng orgs like I'm part of where we go through design reviews to try and cull that sort of thinking, the other less-intentional version is still common enough to be a problem at scale.
I think the general inclination here towards static typing is due to this, rather than any evidence that statically typed languages lead to higher quality software. Engineers just love puzzles. I'm also looking at you, Rustaceans...
runs for his life
Every tool has the potential to be misused in accidental complexity.
But the JS function we would have written would have required someone using it to read and understand it, and the TS function (without using 'any') needs to fully express what its inputs and outputs can look like.
Because of this TS actually tends to guide me towards writing more "Grug brained" code, because I refuse to use 'any' (and throw away TS benefits) and using generics usually requires a trip downstairs for a fresh cup of tea.
The best answer I can come up with is that people just seem to have differently wired brains. For me, static typing- even fairly sophisticated static typing, is simple. It makes the code simpler, easier to reason about, easier to refactor, and with a sufficiently expressive type system it lets you build things in a much more intuitive way than you could otherwise. It’s not about solving puzzles for the sake of them- types remove a big part of the puzzle by letting me explicitly write things down- and letting the compiler keep track of the details.
Certainly plenty of people don’t see it that way, and I’ve heard a lot of people make similar arguments about dynamic typing being simpler and more expressive. I don’t think they are lying but I see a big pile of inscrutable pain when I work in large dynamically types codebases.
I know I’m right about my experience, and I trust other people are right about theirs, so there must be some significant divide in how we conceptualize code that makes one persons elegant simplicity another’s intolerable complexity.
The smartest are those who can make complex problems look simple, with simple solutions.
I have a side-gig as an external examiner for CS students, and well, a lot of the stuff we grade our students on are things that I’ve had to “unlearn” myself throughout my career, because usually complexity and abstraction aren’t going to work out over a long period of time for an IT system. This obviously isn’t something that’s universally true. I do tend to work in non-tech enterprise organisations (or startups transitioning into enterprise) and in this space a lot of what is considered good computer science just doesn’t work. It’s everything from AGILE, where you project processes will trip you over as you try to juggle developers who both need to build and maintain things at the same time. To how we try to handle complexity with abstractions, and how those abstractions sometimes lead to “generic” functions where you need to tell a function 9001 different things before it knows how to perform “because it just evolved”. It’s in everything really, like, we teach students to decouple their architecture and it’s absolutely a good thing to teach CS students, but the result is that a lot of them “overengineer” their architecture so that you can easily swap which database your system is using (and similar) in a world where I’ve never actually seen anyone do that. Anecdotal, sure, but I did work in the public sector where we bought more than 300 different systems from basically every professional supplier in our country, and you’re frankly far more likely to simply replace the entire system than just parts of it.
But how are you going to know that when all you’ve been taught is the “academic” approach to computer science by teachers who might have never experienced the real world? Being self-taught isn’t really going to help you either. I can’t imagine how you would even begin to learn good practices in the ocean of “how to” tutorials and books which are essentially little more than the official documentation on a language and maybe a couple of frameworks.
> The developer's incentive to maximize their own lock-in factor and billable hours are powerful forces
This part, however, I disagree with. Again this is very likely coloured by the fact that I’ve mainly worked in the space where developers both build and maintain multiple things, often at the same time. But I’ve never met developers who wanted to do this. In fact I only meet developers who genuinely want their code to be as easily maintainable by others as possible because we all absolutely hate breaking off from actual development to fix a problem. That being said, I do think there is a natural risk of ending there accidentally if you haven’t “unlearned” a lot of the academic CS practices you’ve been taught. Especially because there is a very good chance you didn’t really “learn them right”.
Regardless of the learnings you had having merit, you don't see a problem with the mental model you've developed, if it's output is giving the same answer 99% of the time?
In itself it doesn't mean it's necessarily wrong, but since you're assessing the quality of other people's work I'd assume you have a heavy bias in there.
At least STEM subjects are tested against repeatable natural phenomena or mathematic validation.
A CPU is indifferent to you complexifying yourself, ad infinitum. It didn't have to be like that.
I suspect this phenomenon explains a lot about how this industry has developed over the last few decades. Any significant software nowadays requires a team of baby sitters just to keep operating. Nothing is ever done and everything keeps changing for no good reason.
Also, I think most developers just don't want to do the same thing twice. And most developers really are writing the same software over and over again through their careers, with minor changes. So they need to change something to keep it interesting, and the only things they can easily change are technology and methodology.
In my experience (including my own development), overly complex designs accrete, as opposed to start off complex.
They usually seem to begin with "This is simple, let me just do this...", then, when we run into Roadblock A, we design in a mitigation, and so on...
Eventually, we have a ghastly chimera.
Other times, it comes from trying to coerce software written for one purpose, into another, and the glue code is kinda messy.
Also, there was an article mentioned here, about "Don't design a general-purpose framework."
I can concur with that. The app I'm releasing now, has a server component that is, in my opinion, way too complex. I designed it, and implemented it, so I get to say that.
The deal was that I originally developed it as a general-purpose framework. It has a layered architecture, and I did heavy-duty unit testing of each layer, as I was writing it.
It works very well, is fast, and secure.
But way too complex, as this app is its only implementation. It handles a lot of stuff this app never touches, like trees of user permissions. I have a very simple, rather "flat" permission structure, so a whole shitton of code never gets used. It was tested heavily, works well, but will never be used. I don't like having unused code paths, but I don't have the luxury of time to remove it (I have removed some, but there's plenty more, where that came from).
If I were to rewrite it (I won't -see "works very well", above), it would be much simpler.
Once you learn to see it though, it's hard to ignore. Now I can tell instantly if code is over-engineered. Unfortunately It seems like maybe 99% of code is over-engineered. The developer's incentive to maximize their own lock-in factor and billable hours are powerful forces. Even developers who appear to be totally robotic and ego-less are often guilty of over-engineering. It works on the subconscious mind. Few are ever able to escape the mindset because they are not fully conscious. They are not thinking about every single line of code that they write. They decide on some goal and then churn out whatever code first pops into their heads to get incrementally closer to that goal... Not realising that, at each step, there were many alternative paths that which were superior.