Meanwhile, he's been promoted to start work on a brand-new Jenga tower since he's demonstrated such remarkable success in the past.
I just want everyone to stop playing Jenga.
Unfortunately as you pointed out, especially in large companies, people can get promoted before the deficiencies of their previous work become clear. This leads to people never learning because the feedback loop is too long, or worse, they never deal with their past code so are oblivious to all its shortcomings and just think they are awesome. Also management tends to reward based on accomplishments today without an eye for costs to be born down the road, which gives perverse incentives to "get it done" programmers who leave mountains of technical debt in their wake for others to deal with.
They believe they go for the latter, but actually they don't. If their code was easy to understand and correct, it would have fewer defects to begin with.
Your second paragraph I totally agree with. I've dealt with such code. Sometimes, I can halve its volume simply by applying local correctness-preserving transformations. That is, without even knowing what the code is doing. I even spotted some bugs in the process.
I won't deny the presence of bugs though, there is endless evidence that bugs always exist.
The system seems to be that it is much more advantageous to your career to rapidly produce gobs of spaghetti -- and confound everyone around you -- than to build elegant code that enables everyone around you. You look better when everyone but you is confounded; you look replaceable when everyone around you can extract as much value out of your clean, powerful code as you can.
That's the real problem with the current model. The data structures will never be perfect and you cannot know how they will change. Yet they do. Then all the FUD from the last migration that scared everyone prevents the team from due diligence and correcting issues when they are discovered. The team waits until the problem comes to a head, management has to be involved, new FUD is created and people dream about perfect data structures to prevent this whole mess.
Remember,
Shipping > Shipping Shit > Not Shipping
I find unit testing does help with all this. It forces exposure of the data stuctures, essentially documenting them. And good coverage gives a list of breakages and sometimes helps find elegant repairs. But I also find myself wanting high level-tests, I guess essentially integration tests, that check not components but overall behavior, and I find writing and maintaining these becomes a real problem.
But I really, really wish I had better tools / procedures for thinking through the problems and designing a proper data model for solving them.
http://www.dodgycoder.net/2012/07/old-school-developers-achi...
"Old school developers - achieving a lot with little"
I'll quote:
> [Ken Thompson] debugs only via printf statements, hardly ever uses unit tests, starts his projects by designing the data structures and then works bottom up, with throwaway test stubs.
Also, Joe Armstrong, the father of Erlang:
> He uses prototypes to solve the hard problems first, and for debugging, just uses print statements. He is a critic of Object Oriented Programming, and favours functional programming languages like Haskell. He never uses an IDE, preferring just Emacs and the command line (no mouse required).
Also, Jamie Zawinski:
> During development he hardly ever uses unit tests, believing it slows things down - he thinks there's a lot to be said for getting the code right first time. In his view, its a matter of priorities, "do you want this to be good software or do you want it to be done next week - pick one because you can't have both".
However, given that at large organizations updates and deployments can easily become political issues, it's a good habit to deploy often. That makes your life easier when trying to deploy new changes because those who are watching or performing the deployments get used to it - and the errors occuring during such deployments.
Requirements desgin, DEMO, Requirements, Design, Implantation, Verification, Release, Maintenance.
The idea was to use a prototyping language for the Demo and then a production worthy language for release. The problem was a lot of Demo's ended up in production for a literally decades because people tryed to create 'over architected crap' which got scraped or takes decades to release.
Honestly, I think the real problem with most early Development strategy's is so few people have a clue how to actually design good software. Great solutions are designed to create minimal systems to solve the problem which are flexible because they are minimal and programmers can alter code, rather than people conceiving every possible change request up front.
If you blindly adhere to some method or other then you're going to find out exactly what the limitations are so you are going to have to be flexible and you're going to have to mix-and-match as time goes by.
As an example, 'agile' comes up here with some regularity. It's a great principle but it's not a religious thing. Feel free to adopt some but not all of agile to come out ahead. Adopt all of agile in a religious fashion and you'll come out a loser.
Read McConnell's 'Rapid development' for a walkthrough of the ones above and then some.
It's why I've still not fully bought in to 'release early release often'.
I prefer to defer releasing for production use until really satisfied with the structures - this way you have no barrier to ripping the foundations up.
If not 100% comfortable with the model - prototype a bare metal improved one (schemaless DBs ftw) - if it feels better start pasting what logic/tests you can salvage from the earlier version and move on.