> I have to remind myself that in "real" engineering there are (in theory) no surprises for the well understood physical processes at work.

That's not true. Tons of sht goes wrong in civil engineering projects. Like the ground moving, or it is not strong enough to support your building, or the workers bailing on you, or suppliers changing the product, or suppliers delaying a shipment, or suppliers going bankrupt, etc...

I think the difference is that it is mostly* redoing the same thing over and over again with minor challenges overall. Whereas in software development you are starting new every time for every new project.