Yet think of surviorship bias, isn't the code most likely to end up in front of a jury the hackiest, most complicated, least understandable, least documented stuff out there?

Think of the VW "scandal" at what point does a piece of code go from being a really effective emissions blocker to being an illegal emissions blocker? If they implemented and trained up a neural net then what piece of code is the violator, the guy who wrote the unit tests the neural net trained on, or the evolved coefficients in the network that humans never wrote, or the manager who told them to write an emissions routine using a NN or the regulators who entrapped them by knowingly writing bad regs with the intention of the .gov selecting winners and losers by selective prosecution?

Richard Bookstaber makes very similar points about the financial system in 'A Demon of Our Own Design', highly recommended.

When the investment banks were partnerships, the partners had plenty of 'skin in the game', and favored stable systems. Going public took the pressure off of the leadership's individual fortunes... and stability seems to have suffered.

They only seem that way to you because you do not deal in finance. A system can only be as simple as the business needs it's supporting whether than be a piece of code or a financial product. Some pieces of code are VERY complex because the business demands it. Same goes for financial instruments.

Is there a real reason for them to be so complex though? "Buisness demands it" isn't really a reason for me. It clearly affects the stability of the system, so shouldn't we look at ways of improving it overall?

Well you can't just say "them." That would be like saying "is there a reason that all that software is so complicated?" There are lots and lots and lots of different reasons for all the different pieces of code in the world. Same thing for financial products.

A finance person would probably say the exact opposite - most deals are just stacking two or three obvious, properly factored/decoupled abstractions together, whereas software people seem (to a finance person) to deliberately complicate things.