I think you have to strike a balance sometimes, maybe bank on your eventual success and put a few % more in upfront to ease the path later. Using something you already know well is good, given you'll be more productive in it to start with and better understand how it's going to impact you in the future.
There will be a point in the product lifetime when you will outgrow the framework and you will need to rewrite large portions of it and pay off the tech debt. Most projects don't get to that point. Once there, it is also the right time to make those decisions that were classified as premature optimization. Now they are merely tech decisions with one crucial difference: at this point you have much more information about the product, the market, the customers, and your team than you had initially, so now you can take the best decisions about what will really increase your velocity.
If speed to market is not essential (such as a side project for fun) then by all means, it's you now, and it's you later, so take the time to enjoy your craft and become proficient at combining the best tools to make your project shine, instead of opting for the cookie cutter frameworks.