Normal software is split into logical modules and layers, which you can understand on their own and then use that to reason about code at higher and higher levels. Many ML models do not offer anything comparable.