> nobody on the business team I report to notices any difference whatsoever, except that feature X took a week longer than it probably would have otherwise

This. And also Refactoring is almost a dirty word so we try to keep it to ourselves and sneak in bits of refactoring when implementing bigger features