You can read all the papers you want. At the end of the day, you actually need to practise writing code! Every piece of software involves a different domain, different requirements, etc.
I feel like the majority of developers these days just copy and paste stuff from the internet and glue some npm/nuget/maven packages together, brush their hands together and feel like gods. That is not how you become a good developer!
How do you become a good developer? Write code _yourself_. Then rewrite it again and again. Keep thinking "how can this be done better, cleaner, more elegantly? how could this be more readable, maintainable?" .. "Maybe others have found a better solution for this, let's do some research and investigate all options, weigh the pros and cons, and select the solution that fits best for this particular problem". Don't forget to factor in the cost of complexity. Is the code you are writing too difficult to understand for the next developer? Is the abstraction too rigid and complex? There is a cost to that.
Rinse and repeat. Sooner or later, once you've written enough code and thought critically about every line, you will acquire a "feel" for what it right and what is wrong. Then suddenly you are an industry expert. Suddenly you are training and mentoring others. Because you put in the goddamn effort.
Even then I think you have to be able to look at your own code critically and want to make it better. The current tech lead in my company churns out a lot of code quickly, but doesn't take constructive criticism of it at all well. It's not easy code for other people to jump into and there are no docs or tests. He thinks it's close to perfect. I don't
I've been disappointed with the industry for the opposite reason: I don't want to be doing this, but this is what is being asked of me.
A current favorite of mine is "The Emperor's old cloths", the Turing award lecture given by C.A.R Hoare. In particular his line "The price of reliability is the pursuit of the utmost simplicity", it applies equally to maintainability and extensibility (I think these are part of what it means for a system to be reliable).
An idea I try to keep in mind while working is not to plan or build for future features but simply leave room for them (meaning don't actively prevent their eventual existence through complexity). It has taken some practice, but it helps guide to a simpler implementation.
Frequently forgotten is the duality of extensibility: subsettability, or contraction. Being able to remove or disable code without rewriting large parts of the application is just as important as being able to extend it!
Especially Microsoft has a tendency to throw out frameworks and helpers that just don’t provide the needed flexibility, and once that has been addressed what’s left is a heap of extensibility points that wrap three lines of useful logic in hundreds of lines of framework code.
So if I could ask one thing of platform developers it is to leave the framework design to the user, and just provide useful primitives. The two hours spent on writing the extra glue code is easily saved on the days spent trying to learn the framework.
In answering the original question, one would hope that focusing on the risks to successful completion should de-prioritize the fixing of things that are unlikely to become problems in practice.
It even lead to Stroustroup saying in a talk that one shouldnt use linked list at all, but instead vectors, because even when linked lists should shine, they do not anymore.
Good abstractions at the correct level and the correct data modelling can make the code around it very simple. Instead of nice designs that are easy to reason about, we have lots of unit tests to "prove" that it works. Instead of making a system reliable and trap errors properly, we use Kubernetes to launch another pods as soon as one goes down. Maybe I am just too old, but I feel we are loosing something going down this route.
(Ok, there is code that's functionality will mean that in inherently isn't simple, but I see far more CRUD apps that are built in an over complicatd manner, than code that is actually doing something algorithmically complex).
Writing perfect bug-free code is nearly impossible and expensive. Hardware is going to fail, network links can get flakey, packets can get lost. Accepting that failure is going to happen no matter what and building for resiliency to it is a far better approach.
Unit tests are along the same line- its been measured that fixing a bug in production is 10x more expensive than finding it in development. Unit tests help ensure that more bugs are found in development, there is less need for, slow, expensive, and error-prone manual QA processes that hold up release frequency.
Unit tests also help document the code to some extent. As teams turnover and new developers who may not fully understand the nuances of some code come in, Unit tests create guard rails that help catch edge cases that may not be obvious to new members of the team and when the test fails makes you go back and give more thought to those areas and why they behave the way they do.
Rereading this, you claim to be on the older side, which I am surprised about- this seems to me like an opinion that someone who is relatively new to the industry may have. I am continually amazed in the new and creative ways things can fail, building for resilience is a 100x better approach that also reduces stress tremendously- Instead of "All hands on deck no one is going home tonight- we just took the system down with this last release and will need to patch it ASAP" the conversation goes to "Ok- we rolled out the release to 1% of our users and we see error rates spiking. Lets roll it back and look at the logs to see what happened."
Similarly with unit tests- it used to be monthly or weekly release cycles, now we are down to multiple times a day because I don't need a programmer who washed out to get around to clicking on the right buttons to give it a green light.
These are tremendous improvements in the state of the art.
> This is about cost optimization to some extent.
Is all the complexity and developer time that is required with Kubernetes really cost effective? We added it at my work and it took around 6 months of developer time from what I could tell (it wasn't me who set it up). Now we have to debug it and none of our team is an expert, so lots of guesswork and googling. Using number of small pods has given us a lot more problems than one big machine. If you are running out of memory because of a large upload, restarting a pod when it crashes doesn't help you.
Kubernetes has lowered our cloud costs and given us high availability, but it has increased development time in a number of ways. Maybe as we get more experience it will get better, but so far I would not say costs have been optimized as a result.
If anything, I think the Greenfield nature may have more to do with it. Today, we take it for granted that software is really expensive to make, to the point where we don't always question the basic patterns that make it so expensive. In those early days, it was far from obvious that software would be expensive.
This is knowledge most engineers I've met completely ignore in favour of the superstitions, personal opinions, and catchy slogans that came out of the '90s and '00s. It's common to dismiss the early software engineering approaches with "waterfall does not work" – as if the people in the '60s didn't already know that?! Rest assured, the published software engineers of the '60s were as strong proponents of agile as anyone is today.
Read this early stuff.
Read the reports on the NATO software engineering conferences.
Read the papers by David Parnas on software modularity and designing for extension and contraction.
Read more written by Ward Cunningham, Alan Perlis, Edsger Dijkstra, Douglas McIlroy, Brian Randell, Peter Naur.
To some extent, we already know how to write software well. There's just nobody teaching this knowledge – you have to seek it yourself.