That's why it's important to start with some testcases of required behavior, and then refactor only code that is supported by tests. Because sometimes the fence is there for a reason, but often it's not.
As an programmer I always try to capture these reasons in my commit messages, after a decade of doing this I’ve gotten a lot of thank you notes years after leaving a project, and I’ve even reminded myself details about my own thought process that I had otherwise would have long forgotten.
One thing I’ve learned by having long tenures is that context change over years is much more likely the reason for nonsensical code than developer stupidity.
The difficulty is making sure that you have all the requirements and aren't missing any, and once a system gets big enough, that gets really difficult.