Ask HN: Cheapest way to host and serve large-ish files?
What options exist for serving large files on the web without getting killed on bandwidth costs?
For example, let's say I want to host a 5GB file that contains an educational ML data set on S3. Assuming 1000 people download it in a month, that would cost me $450/mo in just outbound bandwidth fees. That's almost $0.50 per download.
Obviously there must be ways to serve files cheaper than physically mailing an SD card to someone. What have you done to serve large files cheaply? Is the only option to maintain a server (either colo or virtual) that has better bandwidth pricing?
(
7 comments
| 6 points
| on HN
)
[0] http://academictorrents.com/
Alternatively, there's bittorrent, which was pretty much designed for this.
But considered you can find decently priced dedicated servers with unmetered gigabit ports, seems like installing Linux + nginx and enabling automatic security updates might still be the best option.
BitTorrent would be the natural thing to do, but it has it's own set of issues. Jeff Atwood summarized them nicely in [2]. This was written in 2009, LegalTorrents doesn't exist anymore and you don't need a tracker anymore, but the rest of the points are still valid. His summary is:
> It's ironic that BitTorrent, which is justly famous for equalizing the highly asymmetric nature of most people's internet connections, is itself so asymmetric when it comes to sharing: trivially easy to consume, but awkward and confusing to share. That's too bad, because BitTorrent is such a powerful tool for sharing. Hopefully this post demystifies the process a bit!
[1] http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-ne...
[2] https://blog.codinghorror.com/sharing-files-with-bittorrent/
EDIT: Thinking about it, there is no shortage of free public data sets. I believe many of them are provided by individuals and organizations which wouldn't be able or willing to pay the bandwidth cost out of their own pockets. Maybe you could research the solutions they are using. A starting point could be the [3].
[3] https://news.ycombinator.com/item?id=2165497
But I'm all for paying my own bills as long as they are reasonable. It seems crazy that IA can afford to host huge files for free for anyone but S3 charges so much for so little. I was hoping there was a middle ground. I guess the middle ground is just running my own dedicated server. Thanks for the reply!