Getting the code to work as intended is obviously key - who wants to contribute to a project that doesn't even work? - but my personal view (with no data to support it) is that people are more likely to contribute if they think they can add something to the project, and the speed of being able to come to terms with the project's code base forms a good part of their calculations on how/where to spend (waste?) their spare time.
... Not that it's worked for my project. But then I'm also a poet and my need to make things look pretty on a page sometimes overtakes my need to make sure the code works. All poets know how to "Fail with motherfucking gusto"; we also know how to look stylish while doing it!
Besides, while shipping a working version is crucial, it's perfectly possible to plan ahead and refactor early if you're confident that getting to a working version won't be a problem.
You then need to balance Z with rewriting A to B. The workaround to that is to rewrite A to B while implementing Z but the caveat; is there a time-frame to where Z must be implemented? Because if you implement Z ahead of refactoring A to B you now got to rewrite A to B while not breaking Z while letting Z grow because they want feature Y. Not forgetting you need to recode Z because the old code is old code.
> Write good code. Go back over older code and rewrite it. Then come back later and rewrite it again. Make it better.
IMO, just making it work initially is what matters the most. Ship it, get the bug reports, and fix the kinks. While fixing the kinks, maybe refactor a bit here and there. Some could say, yeah but refactoring makes it easier for others to work on the code with you. My response to that is, yes... but this is a chicken, egg problem. You can write the prettiest code in the world where even a toddler can start hopping on and writing code. But, none of that matters if you never ship, pick up some momentum, and actually have people who want to work on the project with you.