We will never understand software engineering in that way, because we are in the business of automating ourselves out of work. As soon as we understand part of the process well enough that it could become an engineering discipline, we simply build some new tools and let the robots handle that part of the job, moving the humans along to wherever today's frontier of uncertainty happens to be. We will never be engineers in the traditional sense, because that would be a waste of human brainpower.

That seems to imply that there is a surplus of automatable jobs or job functions in "traditional" engineering disciplines, which is ... not really consistent with my experience. Those fields have had nearly as much computer power applied to them as software development, but yet their projects tend to be much better-defined and have a much greater success rate (would you hire a PE firm that had 40% of its bridges fall down?).

Personally I think the high failure rate of software projects is mostly because people on both sides of the equation regard it as generally acceptable, and aren't willing to pay what it would cost to bring software development in line with a traditional engineering discipline, where failure is typically worth guarding against, even if it drives costs up significantly.

You overestimate AI. Incompleteness is everywhere in CS. Overcoming these limitations is not trivial at all.

Besides, software hasn't automated any other engineering discipline, and those are much more straightforward because they're more mature and the principles understood.

I'm not talking about AI. I am making the claim that software engineering will never be "mature" in the sense we ascribe to other engineering disciplines, precisely because we will never completely understand what we are doing; once we do understand the principles involved, we build some new libraries or languages or other tools which automate that part of the process, and we move on to thinking about other things we don't understand yet.

+1 to this - not to say the cliche "AI is the future" statement, but careers span decades and if the fundamental dynamics of writing software changes mid-career you'll be glad you have a strong background in CS fundamentals rather than having taken "Java", "OO Programming", and "Scripting 101" classes.

A strong background in fundamentals is sufficient for a very small number of programming tasks. Do not mistake my top-level comment as an endorsement of the status quo focus on CS minutiae that reigns in this industry.

Your comment is tantamount to claiming an engineer ought to have a strong understanding of theoretical physics fundamentals in order to adapt his career over the span of decades. The reality is substantially different.