Depends on S3's role in your application. If you actually need unlimited, worldwide-replicated object storage with nearly limitless bandwidth, sure, use S3 and pay for it.

Do you just need a place to stash files so your multiple application servers can read them? An NFS export over a private network can very well be all you need, and you can have that file server back itself up to S3/etc at regular intervals.