Yeah, you're totally right here. Maybe a better way to look at this point is whether the code is in the core or the periphery. Give higher priority to refactoring core stuff since many people will have to digest this, as opposed to some periphery item which is stable and rarely changed (assuming you have readability issues in both core and periphery code). A lot of judgements going on as to which code is most important so probably best to have this discussion with multiple team members.
Regardless, my hope was to offer some rebuttals for bad code critics. Getting defensive at every criticism is not productive. Giving in and trying to fix everything is not a smart way to handle criticisms either since you might just be wasting time and money.
In general I'm in total agreement with your points, but in practice this one is really hard to do well. The majority of code that did anything useful tends to stay around for a long time, whether intentionally or not."There is nothing more permanent than a temporary fix."
This is why best practices and established patterns are so important for long term productivity. They help bridge the gap between effort and quality and in general work well. They're not panacea, but the exceptional case is a different issue.