News for hackers. Stuff that matters.
Re:Host Your Own Content by onion2k on Tuesday April 11, 2017 @05:39AM (#14086166)
If the content you write has inherent value, why not host it yourself?

Because of the risk. A 100MB video file that "goes viral" and gets 5,000,000 downloads in a month would cost approximately $30,000 hosted on S3 (calculated using https://calculator.s3.amazonaws.com/index.html). I can't afford that. Obviously I could take the content offline if it suddenly blew up, but instead if I put it on Youtube it would cost me nothing (less in fact; they would pay me.)

Re:Host Your Own Content by TeMPOraL on Tuesday April 11, 2017 @06:09AM (#14086302)
For websites, at least, this can be partially worked around if you stop doing stupid shit. Just don't put 1MB of JS and a 10MB hero image over an article if all you want is to deliver text communicating a message.

Maybe at 29 I'm an old curmudgeon now, but right now I'm building a new website + blog for myself, with a simple, typographic layout, 100% static HTML, and as little CSS and JS as I can get away with. Right now a full article (no images) is about 28KB transferred, including a 11KB CSS framework I'll probably cut in half before even minifying it. The assumption now is that >50KB for a page (excluding images) means I've failed.

Re:Host Your Own Content by theandrewbailey on Tuesday April 11, 2017 @09:46AM (#14087661)
> For websites, at least, this can be partially worked around if you stop doing stupid shit. Just don't put 1MB of JS and a 10MB hero image over an article if all you want is to deliver text communicating a message.

Stupid shit... like hosting a video on S3, or really any static website asset on S3.

Re:Host Your Own Content by llccbb on Wednesday April 12, 2017 @11:49AM (#14098847)
There is lots of discussion about hosting static assets like images or videos, where that _asset/media is the content_. I am wondering what the best self-hosting solution is for these cases.

Let's say I am a photographer and I want to create or share some image galleries. Each image is big, about 10 MB and there may be hundreds of these images (less than 1000). They have a very permissible license, and I want them downloaded and used as wallpapers, for instance. I want the webpage traffic on my site (over flickr) because I want to drum up business and recognition for myself, but I don't want to risk my site going down under hug of death situations. Low cost solutions are ideal because this is a side project.

Is it best to host the main site on a single server (DO droplet) and kick the heavy lifting of image transfer to someone like imgur? What is the cheap and robust solution to this sort of problem?

Re:Host Your Own Content by adrianN on Tuesday April 11, 2017 @06:58AM (#14086520)
In general I agree, but you can't do much when you really want to host a video or an image gallery. Those things are big, nothing you can do about it.
Re:Host Your Own Content by rmc on Tuesday April 11, 2017 @07:59AM (#14086839)
You have a point with JS etc. But what can you do about a video? Only include a 320x260 video in 16 colours?
Re:Host Your Own Content by lelandbatey on Tuesday April 11, 2017 @01:36PM (#14090431)
Assuming video isn't your main source of content, you probably aren't looking to share a very long video. I just ran a test, encoding the first 30 seconds of a larger video into a 720p MP4 with ffmpeg (with CRF of 28), and the output video is 2.9 MB. That's 1 MB smaller than all the content sent over the wire just by visiting the Medium front page.

But, let's say that you're distributing a 10MB video with your textual content, and your post does get lots of attention and suddenly you need to transmit 30,000 copies of that video. Let's also say that you decided to use a CDN to host your larger static content, such as videos.

In such a situation, you know how much that 30000 visitors cost you? According to CDN Calculator[0], it will cost you, on the low end, $18 that month, and if you where using Amazon Cloudfront as your CDN, it would cost you $30 that month.

If you're more price sensitive though, you could easily use a Digital Ocean[1] droplet as your sole content source, and in that case it would cost you a whopping $5 that month, as each droplet comes with 1TB of network traffic per month. If that's not enough you could switch to a SoYouStart server[2] for $40 a month, each of which has a monthly bandwidth allowance of 250 Mbps, which is 78.3 TB of bandwidth per month.

So no, you probably don't need to resort to 320x260 video in 16 colors : )

[0] - http://www.cdncalc.com/

[1] - https://www.digitalocean.com/products/compute/

[2] - https://www.soyoustart.com/us/essential-servers/

Re:Host Your Own Content by onion2k on Tuesday April 11, 2017 @07:00AM (#14086526)
The point still stands. There is a direct cost to hosting your own content that scales with popularity. If your website is slashdotted/hugged to death/goes viral to the point where you go beyond the allocated quota of bandwidth that comes with your monthly hosting costs then you'll have to pay for that traffic. Hosting your content with a business who profits from it shifts that cost, and risk, from you to them. If you don't have much money then that's a really good thing. This is true even if you optimise the size of your website.
Re:Host Your Own Content by kijin on Tuesday April 11, 2017 @08:26AM (#14087021)
Even the smallest Digitalocean droplet comes with 1TB of bandwidth, and I'm not sure whether they charge extra for overages yet.

GP with his 50KB pages would have to get over 20 million views a month in order to use 1TB. Most of us who work in tech can probably afford to upgrade to a larger droplet if that happens.

Hosting your own content isn't free, but it doesn't have to be very expensive, either.

Re:Host Your Own Content by brightball on Tuesday April 11, 2017 @09:16AM (#14087390)
Exactly. The math on digital ocean is strong. I've got a lot of credit with them from a few blog posts that got popular and I still host my blog on a $7 / month Heroku hobby dyno right now with unlimited bandwidth.

The entire reason that I do it myself though is because my blog is what I use for random technology experiments, so I get some value out of that.

Re:Host Your Own Content by loup-vaillant on Tuesday April 11, 2017 @05:44AM (#14086185)
Get rid of HTTP, then.

The web doesn't scale, and that makes it the great centralizer. If publishing ran on peer-to-peer protocols from the beginning popularity would never be an issue.

There is this slight problem about asymmetric bandwidth and ISP-wide NAT, though…

Re:Host Your Own Content by TeMPOraL on Tuesday April 11, 2017 @06:02AM (#14086268)
Would be great.

I've been toying around with IPFS lately, but I still haven't figured out how me getting the repo leads to me joining a global network of peers, with data travelling seamlessly between us. I want the latter, just don't know if it's even supported now.

Re:Host Your Own Content by atrus on Tuesday April 11, 2017 @07:54AM (#14086815)
Unless I'm misunderstanding what you mean, the easiest way would be to just add the browser plugins. Then you can use ipfs just like the 'normal' web.
Re:Host Your Own Content by TeMPOraL on Tuesday April 11, 2017 @08:01AM (#14086853)
What I mean is this: IPFS is meant to enable a p2p data network (web and beyond). How do I become a part of such network after I download IPFS from the project page? I haven't figured this out yet.
Re:Host Your Own Content by pdimitar on Wednesday April 12, 2017 @08:14AM (#14096854)
IPFS serves things on demand. I wanted IPFS to be something like Freenet -- you publish content and the network makes sure to replicate it at least 3 times, automatically -- but IPFS doesn't work that way.

There are some free pinning services (pinning = inform a bot that you would like your content replicated, and then it takes care of it), including at #ipfs on irc.freenode.net, but the network itself doesn't come with such functionality and after I filled an issue on GitHub, it turns out the authors don't think IPFS should do that. Looks to be out of scope for them.

Outside of that, `ipfs add ...` publishes your content's hashes so everybody who knows the hashes can fetch the content from you.

(You also have `ipfs name publish ...` for publishing a website hash to IPNS, which can later be resolved via `ipfs name resolve <peerID>`)

Re:Host Your Own Content by diggan on Wednesday April 12, 2017 @08:54AM (#14097132)
IPFS does not work like that, true. However, Filecoin might fit what you want. It will kind of work like a pinning service where you can exchange pinning content on nodes for filecoins. It'll work together with IPFS.

http://filecoin.io/

Re:Host Your Own Content by pdimitar on Wednesday April 12, 2017 @03:12PM (#14100809)
In my eyes it's a good start. But I am afraid the incentive will bring in malicious actors with time. :(

I'd much prefer something like Freenet's automatic replication model. Luckily IPFS allows it, through its own facilities for extensibility -- if we have the will and resources to invest in that functionality.

Re:Host Your Own Content by irth on Tuesday April 11, 2017 @05:14PM (#14092504)
Well you have to install it. It has a daemon that keeps running, and it "seeds" the content that it has in cache AFAIR.

https://ipfs.io/docs/getting-started/

Re:Host Your Own Content by onion2k on Tuesday April 11, 2017 @07:01AM (#14086535)
That's true, and in the case of video there is an option in the form of Peer5, but you're still going to have a bill to pay if your content becomes very popular very suddenly.
Re:Host Your Own Content by loup-vaillant on Tuesday April 11, 2017 @07:13AM (#14086592)
Personally, I don't. I pay for maximum bandwidth, and if there's a spike it will just throttle. This has its own set of problems of course but at least I don't get surprised by the bill.

They do have "flexible" schemes where the bandwidth can increase as demands grow (of course, you pay accordingly), but that's not the default.

(I'm hosted by https://www.gandi.net/)

Re:Host Your Own Content by kuschku on Tuesday April 11, 2017 @06:10AM (#14086305)
And it would cost you less than $5000 at any normal hoster that has reasonable prices. Less if you buy traffic directly.
Re:Host Your Own Content by adrianN on Tuesday April 11, 2017 @07:03AM (#14086541)
Which hoster provides bandwidth for 50 bucks / terabyte?
Re:Host Your Own Content by gog on Tuesday April 11, 2017 @07:38AM (#14086732)
https://www.digitalocean.com/help/policy/

0.02$/GB which comes out to 20$ per TB and the first TB is included in the price.

Re:Host Your Own Content by kijin on Tuesday April 11, 2017 @08:42AM (#14087111)
100MB video with 5 million views is 500TB. If 500TB costs $5000, it's $10/TB, not $50/TB.

As long as you stay away from well-known IaaS platforms, there are plenty of places where you can get bandwidth for less than $10/TB. Lowest I've seen is $1/TB. Having actually used a few hundred TB with them, it's a bit slower but not that bad.

Re:Host Your Own Content by bitJericho on Tuesday April 11, 2017 @07:35AM (#14086718)
Prgmr can get you to a Tb for something like 10/mo.
Re:Host Your Own Content by Amezarak on Tuesday April 11, 2017 @07:37AM (#14086725)
I used to hit ~1TB/month fairly regularly on my Dreamhost shared hosting, for which I pay $120 a year.
Re:Host Your Own Content by optimuspaul on Tuesday April 11, 2017 @09:37AM (#14087585)
I think you are missing part of the cost. It's not just the bandwidth that would need to scale to support that much traffic.
Re:Host Your Own Content by bluedino on Tuesday April 11, 2017 @08:05AM (#14086872)
A low-end dedicated server can cost you as little as $30-40/month and you'll get 5-10TB of bandwidth
Re:Host Your Own Content by StavrosK on Tuesday April 11, 2017 @06:08AM (#14086293)
Sure, it's harder for bandwidth-heavy protocols, but it's much easier for text. There are even providers who will host that for free (my current favorite is Netlify.com).
Re:Host Your Own Content by z3t4 on Tuesday April 11, 2017 @10:17AM (#14087965)
paywall it and you would become a millionare over night.