Many people do. S3 is convenient though. It handles durability, replication, uptime, etc... and even with the insane markup on bandwidth many startups / companies can afford a $10k/mo bandwidth bill because it's cheaper than hiring some devops and a team to build out the appropriate infrastructure.