I've often worried about this but when I do go back to my old projects it's usually not that bad. Sometimes I'll find something dumb I did and rewrite it, but for the most part the code is easy enough to read and modify.

More than once I've thought I've run into a bug and figure out what the problem likely is, only to go into the code and discover that I already covered that case and the bug is something else entirely.

Most of the rewrites are because I had to do something awkwardly because some feature didn't exist in a library, or it was broken and I had to work around it. Then a couple of years down the road that bug or missing feature has been fixed and I can pull out the hacks and do it the elegant way.