But you can start off with an architecture that requires a full rewrite to make it better.

The number of times I’ve seen this sentiment end in a project being late or never ship is easily 10x more than the number of times a project became crippled later because of performance issues.

Improving performance isn't the only way you can make something "better". A system could be architected in a fragile or rigid way that makes extending it or adding new features extremely difficult without a rewrite.

I can't tell you how many times I've seen people "make it work" and create a mess that has to be rewritten later with much greater difficulty than if even the slightest amount of forethought had been given before hand.