Hot take: The current trend of writing code, AND hiring engineers, is the way it is because everyone thinks they're gonna be the next FAANG-sized company, and need to be able to write FAAANG-quality code and engineer FAANG-quality architecture from the start - with respect to scalability.

Have you seen the personal blogs of devs today? What should be a simple HTML + CSS website with the simplest hosting option possible, is now written in a framework with thousands of dependencies, containerized, hosted on some enterprise level cloud service using k8s.

That's great and all if you suddenly need to scale your blog to LARGE N number of readers, but the mentality is still persistent - when one should be focused on core features and functionality - in the simplest way possible, you're bogged down with trying to configure and glue together enterprise-level software.

Maybe it's a bit unfair to put it that way - a lot of engineers know the various systems and services in and out, and prefer to do even the simplest things that way. But I've lost count how many times I've encountered devs. that BY DEFAULT start with the highest level of complexity, to solve the simplest problems, for no other reason that "but what if" and "it feels wrong that it should be that easy".

So a couple of thoughts.

> Have you seen the personal blogs of devs today?

I don't know that this is a fair comparison, because side projects can and are often a way to explore ideas, understand tech, play around, etc. So I don't know that I'd agree that it's a great extrapolation to the way an engineer works based on side projects or a blog that may have different objectives.

I do agree with the sentiment though, that we want to be watching for indicators to how a team member approaches problems.

> the way it is because everyone thinks they're gonna be the next FAANG-sized company, and need to be able to write FAAANG-quality code and engineer FAANG-quality architecture from the start

I don't know it's fair to say everyone, but is something I agree companies, especially startups should filter for. When I acted as hiring manager, and was trying to build SRE as an example, I would remind candidates, and the team continuously that we're not google. So while we want to bring ideas and approaches in from what google has published as "SRE", we do need to consciously leave large parts out that are appropriate to our needs and stage of maturity.

I disagree that they go down the path they do because they think they’re going to be FAANG-sized, but rather it’s a case of cargo-culting, “we’ll use these tools/architectures because the best companies use them, therefore they must be the best tools/architectures.”

I don't even know if it's cargo culting as much as engineers using their day jobs as opportunities to learn marketable skills for job hopping.

Nearly ever new technology introduced at places I've worked was because someone was keen to get it onto their resume.

I believe this might fall under "resume driven development"

But why are the skills marketable, if no business needs them?

Why are companies hiring people who proudly put Overengineer as their job title on resume?

New and shiny > old and dull.

At least as far as marketing goes, when trying to hire young and hungry devs. More so at startups.

My old job was quite spread geographically and organizationally - lots of small offices with engineers that had more or less total freedom when it came to tooling. It was actually a gov. agency - so that might surprise someone, but it was one of those places that was transitioning to the digital age, and therefore, didn't really have much solid structure.

The various teams pretty much used the tools they wanted to solve the problems at hand - I think we had three different versioning control systems at play, and multiple different databases. Working with data across the organization was a total nightmare.

But we did have a common platform for communication, sharing stuff, and all that. I think we were around 250 devs. and engineers, and a survey shoved that we used over 20 different programming languages.

One thing I DO remember, was that some people in most teams were constantly pushing for the latest (as in 1-3 years old) tools. Someone's writing an API in Flask? No - screw that. FastAPI where it's at. Team x is still writing RESTful APIs? We're doing GraphQL. And that's how it went.

When some of these guys would end up in dev. blogs or being interviewed, they'd of course push the "See, we're not old and stuffy anymore. We've hired lots of young engineers, and right now we're using [trendy stack]"

Clearly there are businesses that serve millions or billions of users, and have a serious need for engineers with experience with the tools to do so. Engineers seeking those jobs are then motivated to use those tools to engineer systems serving orders of magnitude fewer customers simply so they can claim that experience.

I think your general point is true but the personal blogs of devs angle is maybe not the most illustrative one.

We tend to apply industrial strength tools to our personal projects because it's some combination of what we already know, or we're trying to learn or refine an unfamiliar skill.

If you just gave me a linux shell I would not be able to confidently provision a secure webserver for static hosting. But I do know how to write cloudformation and deploy it. Sure this is a personal moral weakness by the standards of HN whatever, but it's where my career has led me so these are the tools I have.

> If you just gave me a linux shell I would not be able to confidently provision a secure webserver for static hosting. But I do know how to write cloudformation and deploy it. Sure this is a personal moral weakness by the standards of HN whatever, but it's where my career has led me so these are the tools I have.

I wouldn't say it's a moral weakness, maybe more of a failing of the tech education ecosystem. It seems bizarre to me that in software, we teach complex high-level things before we teach simple low-level things. Programming students learn very complex high level languages in year 1, and then maybe by year 3 or 4 learn assembly, or what a CPU register is, or how RAM and cache works. It's like teaching a carpenter how to build a high-rise apartment building before teaching them how to measure or use a hammer.

Well I didn't have any formal tech education, just what I picked up on the job and through my own curiosity.

But I mean you don't teach a car mechanic metallurgy and aerodynamics, except to the extent they'll need to apply that knowledge towards specific goals. At some point the discipline is mature enough that people genuinely don't need to, and can't, know every level of it from the ground up.

I think coding is approaching or already at the point where "cs/fundamentals of computation" should be a different degree from "professional software development."

I don't think anyone using Jekyll or whatever for their blog is doing it because they use Jekyll at work.

side note: FAANG is an obsolete acronym according to The Economist, it's now Microsoft - Amazon - Meta - Apple - Alphabet, leading to the new acronym, MAMAA, which has the nice result that we can now talk about the outsized influence of Big MAMAA in the tech world.

I never understand this idea of picking an arbitrary set of language features and saying, “what if your code logic was simple functions and a bunch of if statements”. The complexity won't magically go away, it'll just appear in a different set of problems.

I think it's helpful to divide complexity into complexity in the business logic / problem you're trying to solve, which cannot be eliminated from a pure technical perspective (you should still try to simplify it through discussions with stakeholders though!), and complexity that isn't necessary to solve the problem.

Oftentimes the latter category could be necessary if you were at much higher scale, or if the business evolved in some way, etc., which is where this sort of stuff tends to originate. Just yesterday we were talking at my company about extracting a service in Go, since it's very high scale, very simple, and doesn't change much. On one hand, it's pretty likely we'll need to do that at some point, but on the other, it's not causing any issues right now, so there's not much point in doing it at the moment. Had we gone forward, that would have added complexity for a theoretical concern that may or may not happen in the future.

if this was the case we wouldn't have the problem of AbstractFactory that has plagued the Java ecosystem. if this was the case Golang wouldn't be here seeking to simplify things by not having classes. And having __err__ handling like it does. it's not pretty but it works. I pick on the Java because it's ecosystem is broad. However, the over-engineered complexity that resides there makes you wanna stay away.

One could also use this example for arguing the other way. AbstractFactory pattern would not be needed if Java had more rich feature set to begin with, in this particular example anonymous functions. (Which I believe it nowadays have). Patterns emerge when the foundation isn’t solid enough by itself to stand on.

People needed modularity, DI and callback functions (essential complexity) but since the only way to do that with the language was classes, you had to invent AbstractFactory pattern (accidental complexity).

Everyone does lip service to simplicity, but in reality simplicity is really difficult.

If you have seven conditions driving a decision, a bunch if's might be the simplest implementation. If you have hundreds of conditions, a tree of if's becomes impenetrable. There is no one-size-fits-all when it comes to simplicity.

Some problems are inherently complex. You can't design a payroll system or tax calculation system which is simpler than the set of rules and regulations it has to implement.

> If you have hundreds of conditions, a tree of if's becomes impenetrable.

I mean, it worked for Amazon. I saw the code.

Fair enough, but you probably wouldn't call is simple.

> a tree of if's becomes impenetrable

Even in that case, a tree of if's isn't that bad (it's not great), but far worse is when you have the same set of if statements copied and pasted around dozens of places. Because you will forget to update one of them at some point.

Thousands of classes with indirection is not clean code and write code as simply as possible is tautology. Of course it should be as simple as possible. The interesting question is what counts as simple.

Setting that aside though, the author seemed to mostly be talking about architectural simplicity in the article. He specifically called out "premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs" which I think is spot on. Distributed systems are fundamentally hard and involve a lot of difficult tradeoffs. But somehow we have convinced ourselves as a profession that distributed systems are somehow easier.

The hardest job as a software engineer is to come up with simple and obvious solutions to a hard problem.

Or you can stitch together eight different cloud services and let someone else debug that crap in prod. Not to mention subpar performance and an astronomical cloud bill.

It takes a lot of knowledge, experience and smarts to find simple solutions for hard problems

> write code as simply as possible is tautology

What? That has nothing whatsoever to do with tautology. It's just a statement you agree with. If everyone else agreed with it, it might at most be a truism or an uninteresting statement, but evidently they do not. (They might claim to, but reality shows they optimise for other things - in my experience the simplest work, which does not always mean the simplest code, especially when you're accustomed to the mystic rituals of the Javanese tribes.)

Fair enough, maybe tautology is the wrong word, but I do think everyone agrees with it. Who ever says "we need to have more complicated code"? The question is how do you define simplicity, because it is not always obvious. Every overly-abstracted mess I've ever seen was done in the name of "simplicity". Basically, let's add an abstraction so we can "simply" swap in another database in the future, or handle X hypothetical use case by only changing configurations. Likewise, I've seen 1500 line methods with dizzying, incomprehensible control flow that was nevertheless composed entirely of "simple" if/then/else statements. And a well chosen abstraction or two made things much simpler to read, understand and modify.

Thousands of classes with indirection is absolutely Clean Code. It's in the book.

PipeWire is an example of building a Linux audio daemon on "microservices, architectures that relied on distributed computing, and messaging-heavy designs":

- It takes 3 processes to send sound from Firefox to speakers (pipewire-pulse to accept PulseAudio streams from Firefox, pipewire to send audio to speakers, and wireplumber to detect speakers, expose them to pipewire, and route apps to the default audio device).

- pipewire and pipewire-pulse's functionality is solely composed of plugins (SPA) specified in a config file and glued together by an event loop calling functions, which call other functions through dynamic dispatch through C macros (#define spa_...). This makes reading the source less than helpful to understand control flow, and since the source code is essentially undocumented, I've resorted to breakpointing pipewire in gdb to observe its dynamic behavior (oddly I can breakpoint file:line but not the names of static TU-local functions). In fact I've heard you can run both services in a single daemon by merging their config files, though I haven't tried.

- wireplumber's functionality is driven by a Lua interpreter (its design was driven by the complex demands of automotive audio routing, which is overkill on desktops and makes stack traces less than helpful when debugging infinite-loop bugs).

- Apps are identified by numeric ids, and PipeWire (struct pw_map, not to be confused with struct spa_dict) immediately reuses the IDs of closed apps. Until recently rapidly closing and reopening audio streams caused countless race conditions in pipewire, pipewire-pulse, wireplumber, and client apps like plasmashell's "apps playing audio" list. (I'm not actually sure how they resolved this bug, perhaps with a monotonic counter ID alongside reused IDs?)

I feel a good deal of this complexity is incidental (queues pushed to in one function and popped from synchronously on the same thread and event callback, perhaps there's a valid reason or it could be removed in refactoring; me and IDEs are worse at navigating around macro-based dynamic dispatch than C++ virtual functions; perhaps there's a way to get mixed Lua-C stacktraces from wireplumber). I think both the multi-process architecture and ID reuse could've been avoided without losing functionality. Building core functionality using a plugin system rather than a statically traceable main loop may have been part of the intrinsic complexity of building an arbitrarily-extensible audio daemon, but I would prefer a simpler architecture with constrained functionality, either replacing pipewire, or as a more learnable project (closer to jack2) alongside pipewire.

> we have made writing software complex for complexity's sake.

I think it’s rather that complexity naturally expands to fill up the available capacity (of complexity-handling ability). That is, unless conscious and continuous effort is spent to contain and reduce complexity, it will naturally grow up to the limit where it causes too much problems to be viable anymore (like a virus killing its host and thus preventing further spread). This, in turn, means that the software industry tends to continually live on the edge of maximum complexity its members can (barely) handle.

> I think it’s rather that complexity naturally expands to fill up the available capacity (of complexity-handling ability). That is, unless conscious and continuous effort is spent to contain and reduce complexity, it will naturally grow up to the limit where it causes too much problems to be viable anymore

I disagree that this is something that "naturally" happens. A lot of this thread is about how adding complexity is either a deliberate choice made by software developers or just that the developer simply was never taught how to do it the simple way--both of which illustrate a gap in software development education. When the tutorial about How To Create a TODO App starts with "Step 1: Install Kubernetes", I'd argue we have an education problem.

I’d argue that the fact these choices are being made is natural (otherwise you’d have to explain what the “unnatural” root causes are), and preventing or counteracting them exactly requires the conscious and continuous effort mentioned.

The problem is simple means diffrent thing in a small codebase than in a big one. A bunch of if statements in a code that is small enough to understand everything is ok but when it become big it's hard to understand flow of data.

I do favor simple code but some complexity/abstracion is needed to make it easier to understand

But picking the right abstractions that aren't leaky in any of the aspects you really care about is critical, hard to measure (leakiness isn't obvious, nor what kind of aspects you care about), hard to get right, and hard to maintain (because your abstraction may need to evolve, which is extra tricky).

Obviously, getting that right makes subsequent developments much, much easier, but it's hardly a simple route to success.

I see tech debt and simplicity as a mixture between 'tyranny of small decisions' and each individual coders 'cleanliness' level.

Each individual coder has a code cleanliness level, similar to how every friend's Mom growing up would always remark "Sorry the house is a mess", when it was spotless. If your used to 9/10 and it's a 7, that looks like a wreck. If you are used to 5 and it's a 7, that looks great. I urge other coders to increase their cleanliness level, and to look for others with high cleanliness for guidance. If you are coding next to people that 5 looks good to them, no matter how much they try to pay down technical debt, they never will.

I think tyranny is ultimately showing us that the tooling that we currently have is making is much trickier than should be to evolve those abstractions. Partially this is because of bad abstractions that caused bad tooling and bad tooling that caused bad abstractions. Because it's so difficult, we don't do it. We take the small decision and work slightly harder in a slightly buggier environment to get the new thing done. But of course now the problem is bigger which means its even less likely for us to ever actually pay down that debt.

> “I’m sorry I wrote you such a long letter. I didn’t have time to write you a short one.” – Blaise Pascal

> there wasn't a concept of clean code but to write code as simply as possible

Sounds good "on paper" - in fact, is tautologically true - but it's hard to find two people who agree on the definition of "simple". You say "not having thousands of classes with indirection", and I've definitely seen that over-design of class hierarchies create an untouchable mess, but I've seen designs in the other direction (one giant main routine with duplicated code instead of reusing code) that were defended as "simple".

A lot of complexity comes from premature scaling due to cargo cult or ergonomics.

But I argue a lot of complexity and bugs comes from poor/unclear/conflicting thinking. Especially when it crosses boundaries between multiple developers who had to modify it but didn't truly internalize that part/design of the code.

Bunch of if statements can be described as not simple. Some things in code can only be described as simple. Do those things.