> 1) It's extremely unlikely that you will actually hit a major issue because of scale or wrong software architecture which will bring your system down

that's definitely not the case. In a side project, usually bad architectural decisions are the ones that makes your programming effort more difficult. Refactoring is only a good reason (for side projects) if that will help you ship faster with better quality.