With experience comes the ability to assign create new kinds of heuristics for various problems, which makes it easier to fit them in my head as I work. Reasoning about them, given reasonable simplifications, require experience. Lots of it.
And while acquiring experience you make mistakes, costly ones. You might put lots of time into it without really being able to solve the right problem. It can make you a bit scared of trying it next time. So you avoid it. I think some programmers become so scared of 'doing it the right' because of the risk associated with it that they avoid it more or less all together.
The good programmers are those who manage to get enough experience to reduce the risk before getting scared off by their mistakes.
That's the theory I just came up with anyway.
I've built immensely complex software and in the end the code was good not because I was smart, but because I am stupid, and adapted the code to work around my limitations. It's the "smart" programmers I worry about :)
The key to conquering complex problems is to layer it into levels of abstractions, dividing it into ever smaller parts that does one thing. Then one step up in the abstraction you don't have to worry about the details beneath. If you think about it that's how programming languages themselves work, we no longer push around registers and instructions, its turtles all the way down.
Your mind should always be able to encompass the level its currently on. That's why reducing the amount of code is important. Increasing the entropy in your code makes it more readable and understandable. It also makes it easier to spot patterns. Whenever your code starts feeling beyond easy comprehension it's time to divide it further.
If a problem feels to large, my advice is just to start doing classes or methods that attacks different aspects of the problem, and then compose those.
There's a nice quote by Linus:
"Bad programmers worry about the code. Good programmers worry about data structures and their relationships."
Good structures will shield you from a lot of complexities and simplify your code. Structures are easy to test. Always consider if there isn't a collection or other data structure that could swallow some complexity.
Also, never be afraid of making mistakes, coding is an explorative process and the most valuable thing you can do is to do a mistake and realize it, because that also often means the solution presents itself clearly.
No code is wasted code because even though you might end up deleting all of it the lessons they leave behind are valuable.
In the end coding is a lot like writing a book, get everything down on paper/code without worrying too much and then iterate and move things around until its good :)
The other major trap that I think affects programmers is not appreciating what is happening when you transform your data: when you make a change to data state, how does that affect the system? Where are you persisting that change? Do you need to persist a change or is it simply a temporal snapshot of something happening?
Basically if somebody claims to be a "good" programmer, but they don't have a solid grasp on set theory, they I'm sceptical that they will be a "good" programmer in the long run. Naturally you also have to become familiar with the language you are using and the environment you are developing for.
I guess what I'm saying is that refactoring shouldn't be entered into lightly.
This is a bigger problem the less continuously you refactor. I've seen this mentality in a lot of projects, where they don't dare "mess with the code" because it might introduce bugs or break the tests... but IMHO this is where I think TDD sometimes goes too far.
It should ring warning bells if your code, tests and development cycle is so fragile that you can't do continuous cleanup. In its extreme this is the lava flow anti-pattern.
It must always be preferable to get the code base as compact and simple as possible but bugs will be easier to find and the code easier to understand for everyone... otherwise you're just accumulating technical debt.
Rather than storing nitroglycerin in a pillow factory it usually is a good idea to make it less volatile.
Code generation and config files are evil. If it affects program behaviour, it belongs in the codebase, where you can search for it, and where it's part of the normal release/review/testing cycle.
Much of the coding process is gaining knowledge of the problem you're trying to solve and discovering patterns as you go along.
Also typically features are tacked on over time, and it reaches a point where new patterns are needed that wasn't needed before.
I agree with you in some sense though, some patterns are low-cost and high-reward in versatility & maintainability. Like using events rather than explicit references, and you should use them from the start as a matter of "good habits".
Some more overreaching patterns like factories, modules, plug-ins etc etc only make sense when the code reaches a certain scope.
We live in a time where we're encouraged to make mistakes. Refactoring, fail fast and often... I get the impression sometimes that we can go on failing forever and its ok. Everything is awesome. Because its fun writing code right? Just jumping in there and not giving a thought to the step after the next. Managing complexity? JAGNI. My tests passed, didn't they? And we can always refactor.
Getting it right takes discipline, research and experience. I prefer to get it right first time. Doesn't always work out as planned, but to me the need to refactor is a sign to stop and reassess what brought me to that point. And its always a result of having cut a corner somewhere.
Refactoring costs. And upfront design is about as fashionable as leg warmers.
Continuous refactoring IS focusing on quality, it IS managing complexity. I think you've seriously misunderstood Deming if you think he stood for "Do it perfect from the start". The whole Kaizen philosophy is built on continuous improvement and elimination of waste (redundant code)
I'm not saying do a week/month/year of crappy coding and then "fix it later". Refactor continuously and as often as needed. Writing code is research.
The cost of refactoring is the greatest when you've done it too seldom. Otherwise with the right tools it's pretty efficient these days.
I see larger dangers in doing up front architect astronauting and make decisions before you've internalized all the problems.
For complex project it's of course appropriate to gather requirements, limitations etc and make a rough design, but going into too much detail is setting the cart before the horse... IMHO of course.
We're talking about the same thing.
... or just google antipatterns.
My personal pet peeve is that many programmers don't refactor nearly often nor thorougly enough. It should be natural part of the developing process. Especially with large programs you reach certain points where all encompassing refactoring is needed to simplify the code, yet many power through leading to a bloated mess.
I've found its not uncommon that even decent complex code can be shrunk 70%-80% using refactoring, code generation and config files.