Focusing on business problems is good and all, but if you're ruthlessly outsource everything else, you'll find yourself in the middle of patchy solution with little control and no end of gotchas - technical and otherwise. If you plan to use 10% of a tool and going to pay for 100%, for startups it should be a no-brainer decision.
Surely if you're trying to re-implement PostgreSQL, it's, to put it mildly, interesting (though even here there could be discussions - perhaps you just need a specific slice of functionality?). But for many, many other things, which are at worst a few man-weeks works - and sometimes man-hours, if you're sober with requirements - you could win with the other approach. If you're adding something external, just make sure you're serious about that - documentation, technology background vetting, training, all those things. Otherwise the opinion that "external solution produced by team bigger than ours" is better could be based on an illusion.
Smaller companies should focus on the business problems though instead of rolling their own database, orchestration, or programming language. At a large enough size doing any of those things may pay off but when you are smaller they are just a distraction from your core business.