My currently company only did a specific niche in digital marketing but for sales goals needed to sell outside of that niche. Unsurprisingly, the product didn’t work for those use cases.

This is key. Most engineers want to build frameworks instead of applications. But leave the frameworks to the framework builders unless you were hired to build a framework.

I can jump into a 10 year old rails code base and be productive right away if they stuck to the framework defaults. But invariably someone read some blog posts about how "rails doesn't scale" and built some rats nest of service objects/decorators/portsAndAdapters wtfs and it takes some serious code spelunking to figure out what's going on (hint - the end result is always creating or updating a database record, or making a single api call).

I would argue microservices also encourage bitrot. You can leave some service running on some ancient libraries for years it doesn't need changes. But if you have a monolith where you constantly keep the dependencies up to date - everything stays up to date.

But this in my experience is truly important -- from the very start plan how a new member of the team is going to get really proficient at the system. Ideally, they should be able to immediately recognise/reuse patterns/idioms they already know.

People bitch and moan about boilerplate of Java applications. But when I join the project and I see controllers, models, repositories, entities, services, etc. I am pretty much at home as I know what all those things are doing and how they are connected to each other.

Whenever developers decide to run with their intellectual desires it ends badly.

I try to explain to people that typical business backends applications do not have custom problems. Most of those REST services are just supposed to receive query, run a piece of business logic, translate it into database call and then map response into specified format.

Mostly I’m thinking of more generic services rather than pure ‘business logic’ systems. These are the sorts of problems that Twitter might have in the future. Some of these are problems that Facebook might have gotten to discover in their last big outage.

Many probably noticed that very old, relatively simple machines, cars, tractors, printing presses, etc. can be maintained almost indefinitely. Many businesses do prefer old machines even when there are superior new alternatives and that is because the new versions are just too complex.

To keep maintaining something indefinitely you need to understand the machine enough to know every single piece that needs to be maintained and how to do it properly. Everywhere you need to lube it or clean it or to look from time to time.

Make something complex enough and nobody will have this knowledge. Things start deteriorating. Maybe the data in the database will start accumulating defects, maybe those defects start breaking processes, maybe the processes will not be fixed properly but the errors just get ignored possibly causing other data inconsistencies, and so on.

When you have people who wrote it and understand it fully they can very likely maintain it properly because they understand how to work with it, where to look, how to address problems. Loose them and if the system is complex enough the new crew may not be able to acquire this ability. And then it is just the question of how fast the system is going to be deteriorating.

At the expense of dependencies constantly breaking you, which itself contributes to bit rot by pissing off developers who find workarounds to pin their dependencies and/or switch to work on different products.

I was baffled by this as well for 15+ years until just last year I had a key insight: most developers don’t like to read. In fact, most people don’t like to read. Most people are intrinsically lazy and since growing your skills involves hard work, they don’t do it unless pushed to do so, either by life circumstance or necessity.

Time and time again people have asked me: “how do I get really good at $tech or software engineering?” I recommend a few books or send them my standard list of must-reads. In 999/1000 cases the person approaching me gives me that defeated look that I know means: “Oh. I was hoping you’d say something not involving working hard.” Most people are polite enough not to say it to my face.

When folks have problems with HTTP I ask them if they’ve read RFC 2616, and if not, I offer to bring them a nicely printed and bound copy. It typically goes unread though some use it as a reference. Only once did someone follow my recommendation and read it cover to cover.

Folks seems to lack a kind of basic economic perspective. Spend a week, month, something along those lines growing your skills (or even just 30 minutes reading the docs on that thing you just spent 2 hours declaring idiotic). Yes, this means probably giving up your favorite Netflix binge, video game, or drinking on weeknights in favor of study. But once these skills are internalized they pay back for a lifetime with good returns. Yet for some strange reason people prefer to keep thinking short term and so they never learn what they’re doing. It used to make me sad but I’m making peace with it.

Specs are extremely terse and dry, and I never liked reading them. I pushed myself to read a few and now I find myself craving that kind of technical documentation in my work. The WebAuthn spec is pretty interesting and also has some even more interesting drama on GitHub issues.

My reading list at any point is tied to whatever I am doing in my private or professional life. If my boss asks me to help grow the company my reading list will contain a bunch of stuff about company growth. If I have money to invest but no knowledge of investing -- you will see me listening to investing books while cooking, taking walks, commuting or long distance driving.

Basically, whenever I face a problem I look around for literature to help me with it.

I really hope to have an original problem in my life. Until then I assume that whatever problem I have there is somebody who wrote a solution to it and if I can't find it it is just because I wasn't searching hard enough.

People / Soft Skills:

* How to win friends and Influence People - Dale Carnegie

* Secrets of Consulting - Gerald Weinberg

* Peopleware - Tom DeMarco & Timothy Lister

* Nurtured by Love - Shinici Suzuki

* Turn the Ship Around! - David Marquet

* Never Split the Difference - Chris Voss

http://dhemery.com/articles/resistance_as_a_resource/

http://dhemery.com/articles/managing_yourself_through_change...

Macro Skills:

* Toyota Production System - Taiichii Ohno

* Workplace Management - Taiichii Ohno

* Thinking in Systems - Donella Meadows

* Fifth Discipline - Peter Senge

Agile:

* Extreme Programming, Explained (2nd ed.) - Kent Beck (the annotated bibliography is a goldmine)

* Chaos: Making a New Science - James Gleick

Programming:

* Implementation Patterns - Kent Beck

* Test-Driven Development, By Example - Kent Beck

* Refactoring (both editions) - Martin Fowler

* Refactoring to Patterns - Joshua Kerievsky

* Practical Object Oriented Design - Sandi Metz

* Functional Programming in Scala - Chiusano & Bjarnason

* Growing Object-Oriented Software, Guided By Tests - Freeman & Pryce ("The Goose Book")

* Modern Java in Action - Raoul-Gabriel Urma

* Java Concurrency in Practice - Brian Goetz

* Grokking Algorithms - Aditya Y. Bhargava

Architecture:

* Righting Software - Juval Lowy

* Release It! - Michael Nygard

* Site Reliability Engineering - Beyer, et al. ("The SRE Book”)

* Domain Driven Design - Eric Evans

* Introduction to Permaculture - Bill Mollison and Mia Slay

* Fun with a Pencil - Andrew Loomis

Design:

* All books by Edward Tufte, especially “The visual display of quantitative information”

* The Elements of Typographic Style - Robert Bringhurst

* Understanding Comics - Scott McCloud

* Design of Everyday Things - Don Norman

* Notes on the Synthesis of Form - Christopher Alexander

* The Timeless Way of Building - Christopher Alexander

I can’t tell you how valuable it’s been at times to understand how packets flow over a network. When ECONNREFUSED first rears its ugly head many developers tremble. Over time you learn it’s just a slog of debugging. Emotional control while programming is invaluable.

Also, the realization that people skills are actually no less important than tech skills in a tech setting.

Yes, and this is hard to accept: they are more important than technical skills. I have worked with many truly brilliant engineers who didn't see their ideas implemented because they didn't have good social/conversation skills. The most common version of this is the engineer who gets miffed/irritated/pissed off if people don't accept their ideas after 5 minutes of explanation. Meanwhile, the hustler whose code totally sucks gets the win because they know how to schmooze the business folks (or literally just tell the best lie).

Hard technical skills are not enough for most people to be successful. There are the outliers with "super programmer skills" (and here I'm thinking Linus Torvalds / John Carmack level status) who are just so indisputably expert that they should be heard. But even then they may not get the win. You have to know how to talk to others.

So, the way I think about it instead is that I’m just another type of system that can be maintained, improved, evolved. And my communication with others is simply an interface. And, well, any engineer can grasp the idea that easy to use and well thought out interfaces are valuable and worth working on.

"I think a good place to start is through the service design lens — “This is Service Design Thinking/Doing,” Jon Kolko’s books, and IDEO’s handbook (all available on Amazon). These are all practical, business-oriented works, and will give you the language and tools you need to interact with service designers or conduct your own explorations.

In the tech side, I still think there’s a lot of value in reading through Booch and friends, even if you don’t go on to use UML in a formal way, and even old, deprecated approaches to modeling (think Shlaer-Mellor) can be valuable as an introduction to more modern (and generally more complicated) approaches to domain modeling.

For more academic and less practical works, look into philosophers of language and communication (Rorty, Habermas, Latour, etc) — if you don’t have a background there, try the “Very Short Introduction” series, which gives a good grounding before moving into further secondary or primary literature. What you get from these writers is a way to think about language, communications, and complex systems, so you’re building a conceptual toolkit. (Analytic philosophy, which is another, “mainstream” branch of philosophy, also has applications in the formal comp-sci route.)

Sociology and STS (Science and Technology Studies) kind of occupies a practical middle-ground between philosophy and design; I’ve often said that when we do service design, we’re really just doing folk sociology. There’s no shortage of textbooks, but I’d recommend looking into academic field guides for ethnographic studies (a fancy term for “going out and talking to people about what they do”) and ethnomethodological studies (a fancy term for “going out and talking to people about what they believe”). While the service design books will give you practical tools for conducting workshops and research programs, sociology books will give you a firm grounding on why those tools are used, and how to theorize effectively on what you uncover. I also like measurement theory, which is a branch of psychology concerned with understanding how to define measures and metrics — great tools for clearly defining data and reporting.

Finally, in addition to all this, consider digging into a corporate finance text or two; the three legs of the stool are product and service design; technical and operational architecture; and finance and sales. If you can read your company’s balance sheet and cash flow statements, you’ll be in a much better position to understand where you need to target investments, what the likely return on investment will be, and how much cash (or debt) is available to make things happen. Start building up a good knowledge base in all those areas, and you’ll have the tools necessary to know the “where, how, and why” to apply the “what” of technology to your organization’s problems. "

Especially since reading is great, but actively engaging with what you have just rea, d by trying to solve some related small scale problem, is even better !

I knew a developer who said of such people “they spent six months in the lab to avoid two weeks in the library.”

I think at least to some degree people sort themselves into projects. Typically, the better ones that are also happy and capable with reading other people's code command ability to place themselves into projects that seem like more intellectually rewarding -- which means they are much more likely to form the original team for a new project than to join a project that is already legacy and is not meant to be rewritten.

What strikes me is that it actually is easier to write new code than to properly maintain/refactor old, foreign code. Working with large amount of old code requires a lot of hard, real planning and decisionmaking and learning to live with and adjust to decisions made by others in the past.

This hits "maintenance teams" twice -- not only they face actually more difficult task but they are also less likely to get good devs to help. It does not help that management is usually less likely to give resources to teams in maintenance mode than to teams that are developing new toys.

Right on the money. Everyone loves greenfield because you get to build things up from scratch, try out your own ideas, and there's this feeling of tremendous velocity. You feel like you're just tearing through code, kicking butt, delivering wins. Then the difficult requests start to come in, which usually means feedback from real human users. That's when the rubber hits the road and the grumbling begins.

And like you said, once the project is seen as maintenance it is 'not sexy' despite the fact that all successful software definitionally makes the most money while in the so-called maintenance stage.

Well, that sounds like an utter waste of time. I just wanted to know if this endpoint responded to a GET or to a POST request.

EDIT: to clarify, when I say “problems with HTTP” I mean actually understanding the protocol. Many people for example are unaware of the power of the Cache-Control header and all the things it can do. I used to ask people in interviews what the Content-Type header is for and enough senior engineers didn’t know that I simply stopped asking not to make them feel bad. It’s truly shocking how many people will go decades operating on muscle memory, never bothering to understand what’s actually happening when their code runs.

Thankfully basically everything is utf8 these days.

But I don't think I've read any books on that. What I do is read the code!

I think can do this mostly because I'm interested in it. SW projects die because the code isn't maintained properly and becomes unworkable. So I work on being the counterforce to that, keeping the project alive.

Most engineers are into writing new shiny code, and hate "cleaning up other's mistakes". That's also often rewarded. The results are well known :)

Rather than follow suit, he went in the opposite direction and did what no one else wanted to do… he started a septic tank cleaning company. He said that it was hard work, but lucrative… and that after a while, he woke up one morning and realized that he was passionate about other people’s shit.

AI and crypto are very cool right now. They'll change the world and make some people billions.

But I bet you'll on average have a better career working on sewage plant management software :)

You’re right that there are very few books available showing people how to do this. Usually “Working Effectively with Legacy Code” is recommended but it’s just one book and so can’t cover everything. It’s an area where on the job training and experience definitely makes a difference.

https://news.ycombinator.com/item?id=30651273

They're not about any technology per se, but mostly about exercises in how to tackle complex systems.

(I recently looked into getting a copy of a certain specification printed/bound, but found that the cost would be a lot higher than I cared to pay)

Or, if you’re into arts and crafts, you can get good home binding machines that are not very expensive. It’s actually pretty fun IMHO to punch and bind the paper but YMMV. :)

But I must admit as I get older, I don’t have as much time or patience to devote to this and I’m getting to terms with getting good enough to make a living rather than really understand $new_technology. I think this is partly from getting old and not wanting to sink so much of my own time and mental energy on computer stuff, but also from realizing how little it matters to most PHBs/middle managers who basically hold your fate in their hands. I’ve become much more cynical after a decade in the industry and I’ve realized how much of my own energy and health I have wasted in trying to get a deeper understanding of programming concepts. It was personally satisfying for a while, but it doesn’t pay off for most people. Most orgs want it done fast and cheap, they don’t want squeaky wheels, PHBs think in quarters and only want the thing to not blow up until after they get promoted in a year or two.

Developers who deliver that kind of low cost code fast (albeit low quality) have much more stable careers, better promotions, and get bonuses.

The rest of us ruin our mental health, paint targets on our backs, and don’t get as far. But man do we know our shit.

What I've found is that good companies and outfits do exist out there, but usually they can't pay as well. Or they are boutique consultancies where everybody really cares about the craft...but then you have the realities of consulting to deal with (i.e., dealing with really difficult people).

I am not sure what to say other than: please don't give up hope. Things can get better but you may have to change your expectations or value system to find a place that works for you. Or if you have the stomach for it, try to build a product on your own leveraging your expertise.

For myself, a difficult but necessary change has been not wrapping so much of my self-identity up in 'being a great programmer.' Yes, I know some things in tech, and that's cool, but there is more to life than technology. But just to re-emphasize: I hear you, and it's tough. Wish you the best.

I agree it’s sad but I also have to look out for myself. There is no point in exhausting your energy trying to fix something where no one wants it fixed at best or is actively fighting you to fix it. It’s not even your money that’s being set on fire then, but it is your sanity.

Curious if you’ve found either a better company, or created your own product? It’s difficult to come to terms with mediocrity and needless toil, I went into this industry because I like well designed and efficient systems.

I agree with most of what you said, but this seems a bit ironic.

Your suggestions almost exclusively involve large investments of time with little established proof that they are efficient. Do you genuinely believe reading RFC 2616 "cover to cover" is an efficient way of solving the specific problems they came across?

I would wager most developers wishing to be "really good" actually have a concrete desire like a greater paycheque or employer in mind. If that is true, I doubt reading someone's booklist necessarily is their fastest pathway, and their economic perspective is exactly what stops them doing so.

I think you missed the spirit of my writeup.

But over time this person will be getting better and better and better and at some point they will be able to do their job on the fraction of their time while still keeping momentum and learning more.

This has been my experience.

I have been spending 3/4 of my working hours learning new things for the past about 17 years after I realised this. The actual work takes a very tiny part of my day.

Software development really is a knowledge sport. Writing code is actually pretty small part of the task -- knowing what to write is the important part. If you are able to "write down the answer" so to speak, without spending much time getting there, you can be 10 or more times productive than everybody else, easily.

This. This is how I felt trying to understand legacy core code. I kind of like that though. The idea of being able to improve/rewrite a decrepit piece of code excite me. But you won't be productive, and getting a (somewhat) complete and precise understanding of what it does and doesn't, take a lot of extra time.

But.. that is a dead-end job. There is no career path being the maintainer of a large legacy code-base. It was pretty good if you wanted to rest-and-vest, I could do whatever I wanted. But there were no great features, no demo's, no selling of the work. Even small changes were painful.

Moved to a different team where everything is new and building from scratch and suddenly opportunities open up.

- Was the original team slowly replaced due to cost cutting measures and thusly the people who joined aren't compensated well enough to dive super deep into the system to understand it?

At some point people need to be compensated to care is my theory. Engineers who inherit large systems are either scared or simply don't care, possibly a mixture. But compensate them enough in both salary, time, freedom to understand the code which is clearly critical in maintaining the long term viability of the system.

Thoughts?

If I feel underpaid, I'll do the strict minimum to not get fired and spend my energy looking for another job.

If I feel underpaid, I will do my normal work as best as I can and start looking for another job.

I always try to leave on best terms. You really don't risk anything, you actually improve your possibility of fixing the problem without leaving. And in my case it led to at least couple more opportunities.

Not to mention the world seems small enough and on many occasions I joined the company only to learn I will be working in vicinity of somebody I worked with in the past.

As a fresh developer out of college I had the mindset that frameworks were something you use and they always work and all you need is the documentation. My first job had a lot of "core" code written by another team in another office we would use with exactly 0 documentation. I was trying to get our UI to do something specific but couldn't figure out how to do it by just looking at the functions available, so I ask my manager and he immediately knows what to do, makes 2 small changes (in my defense it was pretty hacky how it was set up imo) and I get what I want.

I remember being moderately pissed because I was already feeling the heat of asking my manager too many simple questions and I just apparently asked another one, so I say something along the lines of "How the hell would I ever have known how to do that, there are no docs available to say that is possible and that is not obvious at all."

He just looks at me and says "You read the core code" and I remember having my mind blown at missing something so obvious for so long. After that I voraciously read almost all "framework" code I could get my hands on, I asked more intelligent questions and my relationship improved my manager and eventually came to be considered a generally good developer.

Once I had this breakthrough I was seeing it in younger developers all the time, and now any time someone asks for general advice on becoming a better developer one of the things I say is "read a lot of code", and more specifically "read the code in your frameworks"