You can miss your deadlines due to dealing with it. When you're on call and you get pulled in to clean up someone else's problem. Bad design patterns cause lots of unnecessary duplication of work.

Yes, you can find other jobs where a mistake brings down the whole team, but it's rarely the kind of mistake that can persist in the codebase for years on end simply because there's no budget to refactor.

Now, you might point out that managers can, in fact, cause even more problems than that, but I don't think you'll find many people who want to work for a bad boss, either and I bet you or someone you know knows people who have left otherwise good jobs because of their boss.