This is exactly, precisely what my experience has been.

Why don't refactor just the parts the need it instead of rewriting everything?

You can do small refactors here and there but usually things get complex after some time. My first advice was to not overthink on architecture in the beginning so it is inevitable that you will end up with something quite unorganized after a while. The assumption here is that in the beginning you won't know what the architecture will look like in the future because you are in startup/discovery mode.

Refactoring such a codebase while keeping everything running can be a monumental effort. I found it very hard to keep people who work on such a project motivated. Analyzing the use cases, coming up with a new design incorporating your learnings and then seeing clear progress towards the goal of a cleaner codebase is much more motivating. Engineers get the chance to do something new instead of "moving code around".

I'm not saying rewrite everything. When you get to this point it usually makes sense to start thinking about these abstractions which I advised to avoid in the beginning. You can begin to separate parts of the system by responsibility and then you rewrite just one part and give it a new API which other parts of your system will consume. Usually by that time you'll also want to restructure your database.