It’s easy to ignore those because we add them piecemeal over time with custom servers, but it is a huge drag on productivity when you add it all up.
Not only that, but you now own these services. If they break, you have to stop working on your product to fix them. Moreover, when your DIY cloud runs into problems, you won't have the benefit of a wealth of knowledge on the issue (internet forums, chat rooms, hiring people, etc) and of course you won't have anyone you can pay for support.
You'll also want a couple of additional environments, and things are probably complex enough that it justifies some infrastructure as code so you can (more or less) stamp out these environments and keep them in sync with your production environment (and also so you can rebuild prod in case of catastrophic failure).
You can definitely get there if you're using your own machines, but a PaaS or cloud provider will get you there a lot faster.
Most startups are not hard tech problems, despite the very strong desire to make them so by a technical founder.
You can even have manually provisioned instances, but you easily spin servers in multiple availability zones and have a load balancer in front of them. Similarly you can easily setup and maintain a database in multiple availability zones without much effort.
Have you really spent a similar amount of time learning your way around deploying applications on the cloud as you have administrating your own server?