This is true, but also in life. I'm so sad I can't just pass my knowledge to my daughter to give her a 30 year head start. I remember telling her 'dont touch that stove it's hot', she nods, and soon after is crying she burnt her hand. I wanted to be mad but it made me realize the one universal truth: advice and information is great, but humans just seem to learn most from mistakes. Made me realize how much time and energy and knowledge is lost, as every human learns a lot of the same things over and over. I'm sure there was a caveman telling his kid not to touch the fire thousands of years ago, too. And here we are. So don't be too cynical and depressed about this cycle, it's pretty natural.
IMO, there's a few ways to try and take a pragmatic approach to the problem above:
1. Make simple ways to do complex things. A library, a language, a framework, they all go after the same thing when designed well: batteries included complex functionality that the end user can't shoot themselves in the foot with. It seems like in this domain (microservices), we are still early and need better support for small companies to spin this up. I've found my current company runs microservices pretty well, but I know it took a TON of infrastructure, support, and education to get it to that point. Few companies can afford to have that.
The danger with this approach is that people who understand the internals become rare, and surface level only knowledge can lead to issues when you get to edge cases. I'll still take it, but wanted to call it out. The linux kernel is a great example of what that looks like down the line.
2. Make as many places as possible for developers in training to fail and make those 5 years of mistakes with low stakes consequences. In some ways, the way the industry hires already does this, but I'm sure there are better ways we can come up with versus the current sandboxes. If your frustration boils over, this is a place to channel it positively.
I think some of the problem is that there's no real standardized training for developers. Many have 4 weeks of 'bootcamp' and are cut into the wild.
And I can't be too negative about it, I didn't even have that. I would just say I learned from 20 years of reading and mistakes.
One thing that's crazy to me is that when people make mistakes, in most cases it's forgotten or swept under the rug. Whereas if they do a full post-mortem and share it with the entire engineering organization, they're teaching everyone a valuable lesson (possibly multiple lessons) for free. If that lesson prevents the same situation from happening again even one time, they'll have "paid off" the first mistake, and it cost them nothing. Free money!
That's not to say it's a waste. Once a person gets bitten, they look for resources and learn from that. But until then, we're basically all stupid apes making the same mistakes in perpetuity.
I don't think this is some sort of innate human thing, I think it's purely down to ego. You think you know more or you're better than the person who made the mistake, and therefore you don't listen to them because you would never do that.
“There aren’t any new problems; only new engineers.”
- Quote I read online a long time ago
Agree with all your points, but there is a third way: apprenticeship with older engineers. I advise new engineers to find their balance between not accepting the status quo but also realizing software is way more complicated than they think.
For example: successfully versioning an API over time while supporting live deployments is (I think) not something you’re going to learn in school.As a further aside... I’ve heard older attorneys complain about newly minted lawyers. They work to pass school and then the bar. But when they start lawyering, new grads don’t know the first thing about how to actually file something at the courthouse. I’m saying, I don’t think it’s constrained to our industry.
That's unlikely to work: our industry has an ageism problem, combined with the low prospects of promotions/raises which results in engineers having to frequently job-hop to get ahead. This results in most organizations having very limited institutional memory. Other compounding problems are a discussion for another day (title inflation: e.g. "Senior engineer" with 2 years total experience)
A few months ago, i engaged to help fix a troubled system. The thing was built around kubernetes and the architecture was basically aligned with how a bunch of contractors were hired. Nobody really understood how it worked and it performed poorly. So poorly in fact that I reimplemented a core component in a shell script, mostly to quickly understand how it worked, and to my surprise, the thing ran faster on my laptop at moderate load than the actual system!
Sounds like a bad IT project, but to be honest whole companies run this way, saving pennies while setting dollars on fire.
Channeling a post from earlier today — I’m not sure if companies are younger and dumber today, but I’m sure that you are older and wiser now than you were then. And I’m sure you are better at seeing the idiocy in companies. Bad projects have always been around. But it isn’t until you have more experience that you can fully appreciate a bad project — particularly when you’re in the middle of it!
I think you have a misconception about what manufacturing is like. Waterfall development is just as unrealistic in manufacturing as it is in software, the scales are just different. People create CAD models of the machines in a factory all the time but those are architectural diagrams - when it comes time to build, the people on the ground (the equivalent of the software engineers) have to figure out all of the details, often using no more than trial and error guided by a little intuition. Experienced individuals know more about the pitfalls and gotchas but they're still creating something new even if they're going off some grand design. Becoming a competitive manufacturer in something simple like ball point pen balls is a years long process of mastering the "craft," gathering real world data, and incrementally improving. Factors like a single critical employee's physical disability can decide the layouts of entire factories so there are no cookie cutter factories for them to build.
The real world is a lot messier than software. Just like we spend most of our time chasing bugs and implementing features, they spend most of their time building or troubleshooting interfaces between machines and processes, optimizing, and so on.
Is it? Aren't a lot of the largest systems build using waterfall, or they certainly were in the past. When i was taught waterfall at university you were allowed to reiterate steps, yet these is a constant straw man that you only gte to move down the waterfall.
CS was focused mostly on the broad underlying structure of computing, things like how software interacts with hardware (this was gutted a year or two before me), algorithms, programming paradigms (functional/etc), the theory behind databases (3rd normal form sound familiar?), and so on. The idea was to give you a really good foundation such that you could self-learn much much easier after college, but also unfortunately that you had to do some of that selfglearning right away because it didn't prep you for getting things done.
I only took one or two ITM courses out of curiosity, so I'm less confident in describing it in broad strokes, but the impression I got is that it was for teaching specific skills while not "wasting" time on better understanding how they work. Very get-things-done oriented. The web development one, for example, used a lot of jquery, but never touched on what was going on under the hood. I can see it working as preparation for being able to do a job, but it left you kind of stuck if you hit a roadblock and didn't know where to look for answers.
If I had to stereotype them, a CS student would spend time writing an algorithm for something instead of finding an existing library, while an ITM student would look for a library and if they can't find one say they can't do it. But the ITM student wouldn't get lost implementing and end-to-end project as long as they could use tech they already know, whereas the CS student has never done such a thing before.
Possibly a case of teaching to the test (interview, in this case)?
My CS education is a couple decades back, but while we had an algorithm course, most credit hours were spent on more practical things like building compilers, building an OS, building a shell, SQL, networking stacks, encryption, software engineering complex systems and so forth.
Is the current CS landscape more algorithm-focused because the FAANGs have made that the only stick to measure by?
[1]they later added a Software Engineering degree which focused more on architecture and designing large scale software system, software project management and so on.
There is a big bubble in tech salaries that is going to get popped as more and more people realize you just don't need a developer for most things and can get away with no-code for the majority of your needs.
This will act as a force multiplier for some, but I think for most developers it will see a correction in their salaries to more reasonable levels.
Most microservices I've seen implemented are not actually microservices. Most teams (unless they are developing "simple" web or mobile applications) have no idea who is consuming their services, or how, or if what they've made is working correctly. They frequently don't have an understanding of the different models of releasing software, much less of a complete system architecture, failure modes, consistency models, reliability estimates, performance limits, etc. Mostly what I see time and again is a team of people who just write some code that seems to work for them, and then go home, without ever considering how or if it's working in the real world.
I don't know anything about modern computer science education. But it appears that new developers have absolutely no idea how anything but algorithms work. It's like we taught them how hammers and saws and wrenches work, and then told them to go build a skyscraper. There are only two ways I know of for anyone today to correctly build a modern large-scale computer system: 1) Read every single book and blog post, watch every conference talk, and listen to every podcast that exists about building modern large-scale computing systems, and 2) Spend 5+ years making every mistake in the book as you try to build them yourself.
It feels like the industry mostly just re-learns the same mistakes over and over like we're in Groundhog's Day (we're the extras, not Bill Murray). But it's equally possible that I just lack perspective and am expecting too much. Maybe the auto industry at the turn of the 20th century also spent decades re-learning the same lessons over and over, as the novelty of mass-producing complex systems continued to elude us. Hell, the new auto companies still don't get it right.