Because software projects fail for people reasons, and people reasons are usually subjective.

If you have a tight team of experienced devs building a simple CRUD app you can do almost anything short of building the thing in Polish BASIC with triple spaced lines, and you'll still get a result. Likewise if you have a team with 8 managers and your best dev has only done a couple of courses on codecademy, no set of engineering approaches is going to save you.

And that's just the factors within the engineering team. For every inexperienced team that doesn't deliver a result, there's 10 teams that are gutted from internal politics or management decisions, and 100 startups that fail to find a market fit. On the flip side of that, every month or so I find either a showstopping bug or a new puzzler of a UX decision in one of Uber's products, but it literally has no bearing on their success as a company. The simple fact is it doesn't matter whether they do TDD, or scrum, or whatever, as long as they don't do anything truly catastrophic.

Basically, the majority of success factors for the majority of projects come from outside the engineering team. And the ones that don't will be dominated by the team's basic development skills, which makes the signal much worse for measuring the effectiveness of anything from a project management, high level architecture, or testing perspective.

Well call me crazy, but imho the reason for that is that for the most part, software engineering is not engineering. We're not fighting laws of nature, we're fighting other people's style and decisions. That's why our work is opinion based: we're basically computer lawyers.

I can't agree with this. We are fighting with chaos, we organize and encode processes, systems and information.

This is the definition of engineering on wikipedia and we fit the description:

> Engineering is the creative application of science, mathematical methods, and empirical evidence to the innovation, design, construction, operation and maintenance of structures, machines, materials, devices, systems, processes, and organizations.

Software engineering may not be an engineering discipline but in theory you could set up empirically studies/experiments to test certain hypotheses of software development practices. In theory you could also review or evaluate design against certain criteria and derive what's better to fulfill certain goals.

And yet, when the subject boils down to people and culture, testing is slow, complicated and unrewarding. Results need a lot of variable elimination and are rarely used for practical decision making. This applies to most “people” subjects.

Testing nature is much easier, and thus establishing norms in these subjects is easier.

Absolutely. And I think there were actually studies on the effectiveness of TDD, and I seem to recall them confirming it.

You are so right, the laws of nature have been abstracted

> we're basically computer lawyers

I like this. I mean, as a description, not so much as a fact, but it is what it is :D

I think the question is easier to answer if it's flipped. Why can't other engineering disciples work fast-and-loose?

The answer is that for software engineering, the cost balance between "working fast" and "avoiding errors" largely works out in favour of "working fast", even with the potential error rates factored in. In other engineering disciplines, the cost of failure is higher (often much higher), and therefore "slow and steady, avoiding errors" methodologies win out in this balance.

There is a good amount of research on software engineering, I love love love "Making Software: What Really Works, and Why We Believe It" by Andy Oram, as it demonstrates the non-opinion-based way of thinking about software engineering. And has some really interesting insights to boot.

Two points.

First, software "engineering" is not an engineering discipline. You might as well call novel writing "literary engineering", but calling it that won't make it true.

Second, the 10x software developer is very much real. People try to ascribe their performance to this or that technology or methodology, but in reality what makes a 10x engineer is a special psychological predilection, not their tools.

Because this is really project management not software engineering. You would get into real arguments about how to micro manage a team of engineers designing a space shuttle replacement. Complex expensive tasks with real tradeoffs don’t have simple answers.

And what is software engineering then?

As I have more and more experience in the software development field, the more I think it is not engineering, but rather art. At least it's a creative process, not always an exact science.

That's because there are too many developers with the title of senior but skills of junior, and a business person can't tell which one is which unless they're one of the few high experienced developers out there.

The only time where a business person knows they hired the wrong people is when people claim for a rewrite. At that point, sorry, it's too late.

Probably because we are dancers, after all.