In my experience, even majorly successful startups that turn into companies worth 10's of billions tend to have lots of technical debt.

Honestly, I believe that at some scale its very difficult to get the caliber of experience and talent that can keep a codebase clean.

People often say accruing technical debt is a form of shortcutting to a more business focused goal, but in my experience the technical debt is almost always created by accident through lack of experience rather than a conscious tradeoff. Especially when you consider a SaaS company where your product lives forever and is meant to scale, taking on technical debt for short term gain is almost never the right decision.

Not to reopen the 10x engineer debate, but I've observed that the top 20% of engineers tend to do 80% of the total work of an org. Of course, 10x is relative... But there are definitely individuals that are 10x more productive than the median engineer. And for whatever reason, that seems more to do with an individual's personality and drive than their level of experience... Though of course experience helps.

Especially as a project scales, somebody who makes good technical decisions will only increase the productivity margin between themselves and others. What I see in a lot of these SaaS codebases is that it takes a week to do something that should take 1 day, due to amount of technical debt and spaghetti code.

Software is still a new industry. I fully believe the companies that win in the long run will be the ones that have the best codebases (assuming software is what differentiates your business). The impact that it has on velocity is just monumental.

Code quality and philosophy of well written code is not stressed at all in school... One example of how far off we are from maturity as a discipline. There is a whole school of thought that's currently neglected that will become mainstream in the future IMO. Similar to other engineering disciplines...

I once saw a customer obsessed junior engineer push out features that our users really wanted, but that none of the senior engineers had time for.

Half a year or so later one of the senior engineers was complaining that he had to rewrite everything that junior engineer did due to how poorly the code was written.

End of the day? The junior engineer's code is what got shipped and made the user's happy. Was it tech debt that had to be rewritten? Sure. Was my team happy with him for being willing to work with us to make the product better? Yup.

Perfect is the enemy of the good and all that.

(I've also seen code where changes took a long time because someone invented a DI framework and layers of XML configuration files when a single if statement with 2 branches would've done the trick. Was the solution well engineered? Yes! Very reliable, good quality code. Took me a few days to get ahold of it and add that 2nd conditional...)

If you had instead hired an additional senior guy who was paid 1.5x compared to the junior guy, it would have been a better outcome for the business all around. Of course just using this contrived example, and the exact salary numbers change the outcome...

The one re-writing touted what they had accomplish, a full re-write with a team of 15 or so people, in a couple months. I can believe that some guys are dickheads and do crap and are anti-social or don't want anyone touching their stuff and think they're the last cookie in the packet.

On the other hand, there's a real difference between writing from scratch something, with features tacked on week-in week-out, contradicting requests, no time to work through issues or problems, being the only one allocated to the problem and having no real plan or feature map laid out, during 5 years and then coming in, when all the system is laid out, all the bugs and domain knowledge have been made visible, there's a working system that can be used as base to understand the objectives, the rough edges, problems and pain points are also known, and rewriting it from scratch better.

Lastly, it's also not known if they had been the ones writing from start with the same constraints if it would have been so pristine as they claim, and it hasn't stood the time for 5 years yet. Maybe it'll end problematic as well but now it took a team of 15 or so that probably cost more than that single guy for 5 years - and still made something that helped keep the business throughout.

Sometimes not knowing something is supposed to be hard is what makes that hard thing possible to do.

Maybe it has to do with being "a new industry", but by now we have accumulated decades. I think it's more because you can get away with it. In other industries, you can't. If a bridge collapses, people die. If a car's brakes fail, people die. If you administer the wrong amount of anesthesia, people die. If you write spaghetti code .. shrug and move on.

Even within the code base of the place I work in, a few hundred sw engs in total, we don't have a shared understanding of what exactly "good", "clean", ... design is. Everybody has an idea, and there are overlaps, but the criteria that people seem to optimize for, and thus the conclusions they draw about design, differ vastly. Within my company, and even more so across our industry.

I'd recommend you check out Code Complete if you haven't read it. It is an excellent intro into best practices although its about 1000 pages and a bit dated. As it turns out, we as an industry do have an understanding of what makes code better, and it turns out, its mostly not very controversial - push complexity as low as it can go, simpler interfaces tend to be better, argument pass through causes errors, functions longer than 80 lines are associated with higher error counts, optimize code for human comprehension later (~70% of the effort that goes into code is maintaining it) etc etc.

> it turns out, its mostly not very controversial

I don't believe this until proven otherwise with actual data. Any programming concept on this planet eventually gets an article on the HN front page that argues for why that particular concept is really bad and then we are having a huge discussion about it.

I don't remember the last time that a piece by Uncle Bob or Martin Fowler—and they're just two examples, I've seen quite a few more who are generally met with agreement every time they're posted—was heavily criticized or disagreed with here.

And when I do see disagreements, they tend to come from people who don't seem to have nearly as much experience as those guys do or who might have misinterpreted their point (much like how the idea of agile has turned into something that only reflects the manifesto tangentially, and accidentally at that) and who are, perhaps not coincidentally, likely to fall in the "programming is easy" camp.

And even then, "controversial in hacker news" is hardly representative of what's controversial in the field at large (which I also understand goes against my own first point).

For example, here are some principles I can strongly justify:

Bubble important info. Structure your code visually such that signal to noise ratio is highest. E.g. appropriately named variables should be more immediately visible than implementation details. E.g. if variable is named well, you can typically ignore the RHS of an assignment... It's an implementation detail. Don't indent such that RHS of assignment is emphasized. We can debate that point, but I'm using objective reasoning for defining why this style is beneficial.

Human readable naming. Name of variables should define exactly what they represent and nothing else. Your code should be written to read as closely to natural language as possible. E.g. if in a functional language, and all functions describe what they do exactly, RHS of assignments are indented to deemphasize them, you can understand high level of any function quickly by scanning LHS assignments and top level expressions. The more unclear your names, the more "mental recalls or lookups" the reader has to do.

Don't use anonymous functions. For anything longer than one line, always used a named function that appropriately represents what it does. Reader should not ever get implementation details pushed into their face. Pushing implementation details to the forefront is the biggest readability error I see in code. It's almost never important for business oriented code unless there's a bug in it. Majority of the time the reader is simply trying to get context and understand meaning of the code. Optimize for this.

Hide your declared functions out of sight. Reader will get high level understanding of their use where called, due to their name. If they want to reference the implementation, they can go to function declaration, but typically they won't need to. Don't declare functions sibling to your core logic. They should not break high level flow of function (does not help readability). Frustrating when people declare functions above where the core logic of the function is... See it a lot in python due to lack of inner function hoisting. Books have footnotes for a reason

Hide implementation details as much as possible behind api boundaries.

Model component apis such that implementation details are not exposed to caller or creator. Don't mix naming of business concepts and render logic. E.g. if you have a generic graph component, nothing within that component should reference your business domain. I see this mistake a lot. More generally, never name something which implies an understanding of a different context than the one you're in. By doing so you're coupling the two domains and increasing the amount of context the reader needs to understand your code.

Prefer immutable variables. The more constants and immutable state you use, the fewer things you have to track in your head as you follow code. You know once you see an assignment, that variable will never change. You don't need to scan every line between assignment and later use to determine whether that var is later modified.

Don't nest expressions too deeply, such that it's difficult to parse. If you can instead assign to appropriately named variable, reader can typically ignore the expression altogether.

Prefer named function or variable that describe the result of a computation rather than inlining an expression. If featureFlag == 1 represents isFeatureEnabled, assign that to an appropriately named variable. Anytime you make the reader "infer the name" of a variable by parsing the expression themselves, you make your code harder to read.

camelCase is superior to snake_case because a variable represents a single concept, not separate concepts represented by a string of words. Adding underscore unnecessarily makes words visually distinct when you're dealing with a single concept. They also add verbosity and length that doesn't benefit anything. Is there an objective argument for snake case being more readable other than it's the convention in some languages (not objective reasoning) (Flame war go!)

Anyway, I could go on with a number of more points. Am I suggesting there's a correct style? Not at all. But you can absolutely use strong and objective reasoning for why one style is superior to another. I don't see people typically apply this kind of rigor to their style, they tend to just prefer one approach "because".

Different people will weight things differently, so can come to different conclusion given same evidence, but there's absolutely a whole set of philosophy and logic you can use to justify style, and it's not explored at all in academia really.

Final note. There is definitely an element of cultural bias as well. People who tend to read code of one style will more readily be able to parse that style. There is no universal truth. But if we all start from the same state, we should justify which styles are best with strong reasoning.

snake_case, is obviously the more readable. In fact your point shows this cognitive dissonance we all have to a certain extent. You argued (and I agree) with a lot of details that are important for readability at "first sight", "intuitive layout" etc, and then you go to say camelCase is more readable...

We're not even taking into account people with small amounts of dyslexia in this, but

certainlyThisIsNotMoreReadable than certainly_this_is_not_more_readable

Is it? :) gzip will take care of the underscores

It took away from my overall point, as I don't have strong convictions on that one.

That one is much more subjective than the other points!

Point is that we don't have a shared, common, well-defined core in our industry. A list that everybody adheres to. Our lists are not in sync.

The variety of programming languages that people can choose from now does not help either. Standard practice in one language is a cardinal sin in another. Different patterns, different principles, different paradigms. People mix and match freely.

I provide objective and concrete reasoning for why these style elements are beneficial. In the wild I rarely see people justify their style with strong convictions.

Again, different people can reach different conclusions given the same evidence, so there will never be a universally correct way to do things.

But there's also is a lot that's done by convention that's hard to justify objectively.

The biggest faux pas I see is pushing implementation details to the forefront. There are arguments to be made that this can be beneficial in some contexts, such as algorithms where you want a "single pane of glass" into what's happening. But you can also objectively specify those contexts and when a style is more applicable to one context vs another.

So long story short, no, there is no best style. Yes, you can objectively quantify why a style is good or not, and in which contexts. Beyond specific style elements, there is a philosophy underlying the readability of code style and will eventually become a big part of education in software IMO.

Other industries/areas have firmly established standards. Our industry needs those too, badly. Not just for style, but overall for design and architecture.

Maybe it's because software development is so easily accessible that we don't have them yet. To become a surgeon, you need to go to med school for an eternity. To become a construction engineer you need training and government certification. Both areas move slowly, compared to programming. In our industry, a language that's older than 10 years is called "mature" and the hipsters leave it for a younger one. And the next 16-year old can jump right in. That's not bad, it keeps our industry exciting, but it does not allow us to establish rules everybody agrees to, applies and defends.

I think this was one of the most disheartening realizations of my career. I spent and still spend a lot of thought on how to improve code for readability and maintainability just to realize companies can make a lot of money with crap for a long time.

All else equal, the company with the better codebase will be more competitive/make more money.

But at the same time it's just true that for most companies code quality and technical excellence are not core qualities of their culture. A lot of companies say that it is, but they don't live it... Or the original founder/technical team don't really have the chops to enforce that quality and make it a core element of the business.

Of course that doesn't meant they're not smart. I've known a huge number of highly intelligent people that don't care about code quality, or the future human reader. Needs to become embedded in culture of the org for it to trickle down and become lived in practice.

I have also done the opposite of taking an existing monolithic spaghetti code system and start isolating functionality without changing behavior. This then allows to improve and replace sub systems as needed.

Until I can hold up a plaque where humanity agrees that I'm a proven 1x, 5x, or 10x engineer and I deserve money and a job according to each of those levels, it's all just posturing and debate.

The traits I see in engineers who I consider to be 10x:

Growth mentality. Always try to do things the "right way" rather than just completing a task. Educate yourself via internet/books to understand all points of view on an issue, and tradeoffs of each approach.

Setting a high bar for your work. Never solely focus on getting the job done, always question and focus on what the best way to get the job done is.

Extremely self critical. Don't get attached to your work. Always criticize yourself and question whether decisions you made were correct.

Objectivity. Always quantify objectively to yourself why you made a given design decision, why did you name your variable X vs Y? Could you justify the merits of your approach to others without relying on subjective points?

Focus on business value. Optimize for what drives business value. Don't pursue projects that are technically interesting that don't provide long run value, for example.

Work with intensity. These people legit code 8-10 hours a day straight. They aren't coding for 30m then browsing the internet.

Never give up. Regardless of how technically challenging or impossible a task seems, they will work tirelessly to find a solution.

Self managing. Can operate 100% independently without constant manager intervention. High level direction still important to align on business goals of course.

Ownership. These people take pride in their work and feel a sense of ownership over their code. They don't need to be asked to step in when something they've worked on has an issue.

It's not about specific knowledge at all, or training.

People could learn to follow these principals, but it's much easier when it's in your nature. I have not seen anyone really change their trajectory from median to superstar before, but I'm sure it's possible with a mentality shift.

That is actually a failure mode - the engineer that tirelessly works towards an impossible goal.

The 10x engineer has the critical heuristics/intuition/skill to avoid dead-ends, and also the engineering taste to concentrate on technically hard but possible tasks and the skills to deliver.

I more often see junior employees give up on tasks they think are impossible that aren't.

That's physically impossible, unless you're performing some assembly line task like creating a frontend prototype from a marvelapp mock (tons of html, css, js).

Can you do creative work for 10 hours in a stretch, every day.

> One afternoon, I was bent over a program listing while Wendl was staring into space, his feet propped up on his desk. Our boss came in and asked, “Wendl! What are you doing?” Wendl said, “I’m thinking.” And the boss said, “Can’t you do that at home?”

Averaging around 350k lines of code a year for a few years. This was mostly frontend/product focused work, so there is more boilerplate than in other disciplines, of course.

It's also important to note, the higher quality the codebase is, the less energy it takes to add to it. If you spend 50% of your time reverse engineering spaghetti code, it will be way more tiring than adding to a codebase where design is sound, code is very human readable/friendly etc.

It certainly takes a lot of energy, but why do you say it's impossible?

but I should point out that long stretches of time are different from quite (context switch free) streches of time.

I have worked on less clean codebases where I spend hours tracing state through spaghetti-like code, and there's no way you can productively code for lengths of time in those environments.

It's always better to refactor that kind of stuff ASAP, assuming the longevity of the product matters, and the refactor is a tangible improvement over the original.

How to judge whether a codebase is high quality?

Consistency of design and abstraction. Does the code use the same abstractions and design throughout? Consistency is one of the most important things in quality. How many unique concepts and abstractions do you have to learn to understand the code? E.g. in frontend context, do you use the same pattern for managing business oriented state? How is that state updated? Are those patterns consistent throughout your codebase? The biggest problem I see with the spaghetti codebases is they lack consistency in design.

Separation of concerns. What's the blast radius of any given change? If I change render logic, can it impact business logic, and vice versa? A big problem I see is that codebases do not appropriately separate these type of things out. Meaning the code is fragile and it's easy to introduce bugs inadvertently.

Anyway, I can list many points, but I'm on mobile now.

High level indicators would be: How quickly can new devs jump into the codebase? Is it easy to diagram and explain to others how the code is structured? How many bugs come out of your codebase? How quickly can you add new features to that codebase? What is the "blast radius", or is it possible to break something seemingly unrelated to a change you're making?

How do I know my codebase was good quality? I was extremely self critical when writing it and ended up rewriting it a few times before it was in a state I was satisfied with. Number of bugs are extremely low and feature velocity is high, relative to other areas of the code.

Say I had all those traits and you decided I was a 10x and now I've got my shiny label and I'm happy.

Then I come across someone else with a stricter definition and more requirements. They say I'm not 10x. Who do I believe?

Because there's no widely accepted standard, it just ends up being a game of finding the right people who like the qualities that you possess.

But the truth is there are people who are insanely productive relative to median, and 10x is just a rough term to refer to them.

I do think it's worth paying those people 2x or more over paying 1x for two median devs. The trick is being able to identify them accurately... that relies on a well honed interview. There's risk in compensating people so highly if your judgment turns out to be wrong.

Certainly definitions won't be universal, but this type of person tends to do very well regardless of the environment/language etc.

No, I'm not talking about the BS certifications where you can spend a few hours reading a book and pass a multiple choice test... but something more "elite" and well rounded to give a quality stamp of approval.

It would be a great thing if candidates could go through one process, get that stamp of approval, and receive offers from X companies.. rather than interviewing 100 times. I understand the needs of companies are individual, but most candidates who are "strong" will do well interchangeably at these companies... if it's more of a generic development role.

The difficult thing is that the industry seems very stuck on algorithm and design problems as the sole focus of judging a candidate, which are pretty easy to game (design less so than algorithms). To identify these kind of people you really need to judge real world code and productivity... large projects with more elements of design.

Beyond technical skill, how hardworking is the person? How much will they care? Personality traits like these tend to matter a lot more than specific knowledge.

Anyway... this is kind of a meandering response, but I do believe that in the future we'll have a much better system for interviewing with less redundancy and higher accuracy. The goal is to maximize correlation between performance on the interview and performance "on-the-job".

There are so many counter examples (Facebook, Microsoft to name two) to your point it simply can not be true.

When I say long run, I'm talking hundreds of years. Software as an industry is not even one generation old yet. There are countless advancements that will come.

That being said, I don't consider social networks to fall into the class of an area where your product is the main differentiator. Network effects come first, product second.

Database products are a better example. JIRA another. It's the most widely used now but extremely slow and buggy. I sincerely doubt JIRA will be the winner 100 years from now if they don't improve technically.

In fact, a lot of software companies these days are just competing with traditional companies, but with better software. That's the whole selling point. E.g. fintech replacing banks, uber replacing taxis, insurance companies that make it easy to make claims etc. The bigger the tech disparity, the bigger the advantage.

Lets see!

I suspect that there's almost nothing going on at the Ford Motor Company that has even the remotest connection to anything technical that happened during the early years of the company's founding.

Sometimes, stuff just really does age out.

Engine blocks are still cast, sheet metal is still bent, pressed, and rolled so I think there is a lot of the original years still around. Yes, I’ll give you that there have been huge advances in sophistication, but it seems to me the early years are still present.

>technical debt is almost always created by accident through lack of experience rather than a conscious tradeoff.

I agree with this, at least early on in the business. But my experience is that many organizations eventually realize their technical debt. Sometimes the realization comes so far down the line that it feels like a monumental task to fix and then it does become a conscious decision. “We know the code base is shoddy and needs to be fixed, but the schedule/cost risk outweighs the quality risk.”

>Software is a new industry

In the early 1990s I heard it referred to as the “cave drawing days” of the industry. I wonder if it’s similar to how mechanical systems were in the early days of the industrial revolution. Lots of innovation, very few codified best practices related to pressure systems etc. It seems like the steam era of “move fast and break things”. Eventually, industry caught up and put together standards like ASME codes (or were regulated to do so). I’d like to see this more commonplace in software beyond its current adoption in a few industries

In terms of your example, yes. The problem with software is that a bad architecture can complete hobble it and there are plenty of cases with unbeautiful code that makes a pile of money. I think there is artistry in software, but that wasn't the primary reason I compared it to music. I was trying to express the vastness of the domain and the range of complexity and how one could find a part in the search space that is simple enough to be called easy. Hence the garage band metaphor. Not everyone is going to be writing compilers. Some people don't want to throw their brain at a heap of complexity all day long. They just want to make pretty buttons that have a satisfying snappiness to them and call it a day.

What I'm trying to say is that there isn't one "programming" and for the people that need to hear "programming is easy, anyone can do it" in order to get the courage to try to take it up, I think that message is good. It doesn't mean they'll be working for Nasa on space probes, but they can make a comfortable living on low stakes stuff and have fun while they're at it.

One can argue that White Stripes' 7 Nation Army or Andy Warhol's Campbell's Soup Cans are not feats of technical mastery by any stretch of imagination, but the key is that there is no correlation between technical difficulty and monetary value in the first place.

With that said, it's also worth mentioning that the world is vast enough that there's a bit of everything. Yes, there are companies making millions on the back of terrible excel spreadsheet hackjobs. But there's also Google's impressive infrastructure. And there's lichess.org running a hugely popular app on just a few servers. Success in itself it not one-dimensional.

I see arguments like this a lot and it think it conflates two very different kinds of "technical difficulty"—design difficulty and execution difficulty.

Given the sheet music, any guitarist can play 7 Nation Army in a few minutes. It is not physically difficult to play. Also, it's not hard to come up with a completely original melody or artistic work. Just throw a couple of random unrelated things together.

But it is extraordinarily difficult to have such a mastery of composition and understanding of what music is already familiar to people to be able to find a melody that is both original and enjoyable to a wide variety of people. The level of cultural, musical, and historical knowledge required to do that is extremely difficult and rare.

It's like taking your boat to the world's most popular fishing hole and knowing where everyone else fishes so well that you can still manage to stake out an unvisited spot that hasn't been fished out and reel in a giant.

The problem with this argument IMHO is that if it were true, one would in fact be able to just bang out hits by developing such mastery. But music hits get popular primarily on the basis of luck, survivor bias, branding, etc, not strictly based on artists' composition skills. There are plenty of very talented one-hit wonders, and even criticisms that popular music is "manufactured" (implying that it is relatively easy to just follow some vague formula of 4/4, II-V-I chord progressions and lyrics about love and end up with something that resembles a hit).

Isn't this exactly what Max Martin has been doing for last 20 years?

But given how many musicians and producers exist and how utterly small is the percentage of people who are comparable to Max Martin, I think it's reasonable to speculate that his success might be partially/majorly attributed to factors of luck/survivorship bias/branding/popular-music-as-a-mass-produced-product I mentioned.

If you have 25 programmers on a project, it is rather likely that 4-6 of them do at least half of the work. That would be a 5x developer.

Very often, productivity is linked to perseverance. If you socialize and only actually codes 1-4 hours per day, you likely are far less productive than the on-the-spectrum programmer who pounds out code for an entire work day (or longer).

The performant developer solves problems quickly, without creating problems for others, i.e. he unloads burden from the project.

This, essentially, ends up with a PR review that takes hours to complete, only to end up with at least a few "changes required". Leading to updates, and a re-review, usually with enough time lapsed that the review ends up being "from scratch" (partially due to the size of the PR, partially due to the time lapsed).

We've now mostly converged to "start with a PR just setting the skeleton up and one testable function", meaning that what used to be a single monster PR now ends up as 5-15 PRs, each one much easier to review. Amusingly, this also tends to end up with things being completed faster.

The knowledge of thread safety is not an inborn thing nor issue of talent. It is literally issue of knowledge that you can acquire if you previously did not had it.

And companies can get away with bad code for quite a long.

I've met a lot of folks who didn't believe it... right until they met one.

> That application got the company to IPO

They got lucky they could IPO before scaling became an issue.

Just imagine Google not being able to scale. Or Facebook.

That right there is the truth about 10x engineers. We have one in particular I will mention. Sometimes he is a 5x, and at other times he is 20x. This is true and measurable according to the number of tickets that get closed, stay closed, and test as correct. He doesn't read HN, stack overflow, fb, twitter, tiktok, blogs, or emails during work hours. He sits down and cranks out code.

10x engineers also have compounding effects. These are impossible to measure, but you'll notice them when you see them. They flat out write better code.

Better code fails gracefully, is easy to read and especially nice to extend. Juniors seems to magically be able to ship features really quickly on that particular codebase and onboard really fast. If you dig deeper you'll see the 10x giving hints and feedback.

Take a close look at various bridge designs, and you'll see the artistry.