News for hackers. Stuff that matters.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ilaksh on Tuesday March 04, 2014 @09:23PM (#7344844)
The thing is there are many providers who can do the same and most of them will do it for less than half of this. Some less than 1/5th. I think they should move this to Digital Ocean and save $5000.

The bias towards AWS for this type of application is ridiculous and a big waste of money.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ghshephard on Tuesday March 04, 2014 @09:36PM (#7344879)
Whenever anybody makes this type of statement, I'm alway interested in knowing if they've ever run a site with this type of traffic, and this many customers.

In particular, have you ever run a site that consistently serves over 25 Terabytes of traffic/month, or have you worked with someone who has?

I guarantee you that no company I have worked for in the last 15 years, could have ever run this type of infrastructure for $7K/month. Its absolutely amazing.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by Zarel on Wednesday March 05, 2014 @02:57AM (#7345902)
My site serves 25 TB/mo, and it costs me $80/mo...

$60/mo for a dedicated server, $20/mo for CloudFlare. The dedicated server only serves 1 TB of it, the other 24 TB is static assets cached and served directly by CloudFlare.

Here's a screenshot of CloudFlare Analytics for the last 30 days: http://d.pr/i/6Z8S/5GU2Ni8t

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ghshephard on Wednesday March 05, 2014 @03:20PM (#7349589)
Thanks - that's eye opening.

So, what this really comes down to (after a good nights sleep) - is what type of traffic/transactions are you running on your back end infrastructure.

If the data is static, then you can probably (these days) cut your costs for 25 Terabytes/month from $8K to $800 (or, in your extraordinary case, $80), simply by being a bit intelligent as to how you make use of VPS/CDN/CloudFlare Transfer allocations.

On the flip side, if much of the data you are transferring out is the result of dynamic back end transactions, queries, and generation, then it's unclear to me that you can (easily) recognize the savings that you might see when generating static content.

I'm interested in knowing if CloudFlare will start throttling/shutting down people who pay $20 and use 25 TBytes in the long term though - that alone, for some organizations, will cost them more than the extra $8K they would pay to AWS (who, have zero problem with you using 25TB, 250TB, 2.5PB, etc...)

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by Zarel on Wednesday March 05, 2014 @03:56PM (#7349831)
Yeah, I'll admit that other CloudFlare customers are likely subsidizing the amount of bandwidth I'm using.

Funny thing - back when I was using 10 TB/mo, my site was hosted entirely on DreamHost's $9/mo shared hosting. I moved mostly because I was starting to get several hours a month of downtime - presumably, they were gently nudging me off their service.

I've seen plenty of $60-$100 dedicated servers come with unlimited-use 100Mbit connections, which work out to 16ish TB/mo before you start getting to 50% saturation. Of course, those are still subsidized in that that pricing is possible only because most people who buy it don't max out a 100Mbit connection.

Still, though, S3's 9-12¢/GB bandwidth pricing seems a bit high. Bandwidth at DigitalOcean (presumably unsubsidized) is 2¢/GB, which comes out to a much more manageable $500 for 25 TB.

With dynamic content, CloudFlare has Railgun, which takes advantage of the fact that dynamic content is usually mostly static. Still, though, if you have 25 TB of dynamic content, I presume bandwidth stops becoming the limiting factor in your cost of operation.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by hackerboos on Wednesday March 05, 2014 @07:59AM (#7346658)
CloudFlare cannot be making money on you as a customer. $20 for 24TB is too cheap.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ceejayoz on Wednesday March 05, 2014 @09:40AM (#7347140)
It's offset by the many customers paying $20 and using 1 GB, I'd imagine.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by gaadd33 on Tuesday March 04, 2014 @10:01PM (#7344981)
I'm not sure if RubyGems gets more traffic/has more intense computational needs/has more users than OkCupid, but that used to be hosted for about ~2-3K/mo from what I recall. However, that's not amortizing the cost of the hardware.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by toomuchtodo on Tuesday March 04, 2014 @10:22PM (#7345067)
> I guarantee you that no company I have worked for in the last 15 years, could have ever run this type of infrastructure for $7K/month. Its absolutely amazing.

True, but lets not compare offerings of the past to now. There is still room for practical efficiency gains.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by adrianpike on Tuesday March 04, 2014 @09:56PM (#7344959)
Are you including person costs in that $7k? If so, I totally agree.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ghshephard on Tuesday March 04, 2014 @11:02PM (#7345191)
Only the person costs associated with the configuration/management of the servers - not the people time associated with the code, and high level system administration (which you still need with AWS).

I.E. The people who bought the servers, racked the servers, went down to the CoLo at night, set up the virtualization environment, hooked up the routers, configured the routers, the switches, the firewalls, the vlans -- those people I am including.

I'm not including the DBAs who manage the schema, people who push the code, do the design, etc...

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ilaksh on Tuesday March 04, 2014 @09:49PM (#7344935)
Have you worked for Rackspace, Linode or Digital Ocean?
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ghshephard on Tuesday March 04, 2014 @11:18PM (#7345244)
I've worked for one of their (direct) large competitors, but haven't worked for those three companies.

I've currently got active accounts with all three of those VPS providers - I love them, and use them every day - particularly Linode, but also Slicehost/Rackspace, and DigitalOcean. I even have a bare metal server at ServerBeach - which I realize I need to shut down...

At this exact instant I have six terminal windows open across DO/Linode. I host a moderately popular California Food Blog, and have about 15 years experience in various companies that have had hosting responsibilities.

I'm not saying you can't do great things with the VPS providers - I'm just suggesting that the tradeoff between saving $2-$3k (at most) with Digital Ocean, would be more than made up by the technology risk, hassle of having to re-invent a lot of the services that you get automatically from AWS.

That could change sometime in the (near) future - but right now, AWS is an easy (and honestly, all things considered, relatively cheap) solution for this type of application.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ilaksh on Tuesday March 04, 2014 @11:56PM (#7345370)
But they are just using Amazon's Cloudfront. They aren't using SQS or anything.

What technology risk is there in setting up Varnish and nginx on Digital Ocean? Or better yet some kind of out-of-the-box open source CDN. You would save a lot more than $2-3k.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ghshephard on Wednesday March 05, 2014 @12:46AM (#7345530)
I'm referring to Digital Ocean's technology. AWS (and Media Temple) had it's teething pains as well in it's first 4-5 years. I remember some fairly broad outages with their back end storage - but they've mostly dealt with those, and, the risk has gone down.

Note - there is another option - Deploy on multiple Platforms and be smart with your DNS balancing (http://www.dnsmadeeasy.com/services/global-traffic-director/) when serving content. Particularly now that Digital Ocean is in Singapore/Amsterdam/NewYork I can think of some useful things I could do with $10/month droplet (2 Terabytes of Transfer) $300/month, in theory, gets me 20 Terabytes in Asia, 20 Terabytes in Europe, 20 Terabytes in North America. Now, whether DO would shut me down if I actually started using that Transfer is another question altogether...

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ksec on Wednesday March 05, 2014 @02:11AM (#7345785)
Last test i tested DNS balancing ( from DNSMadeEasy )sometimes doesn't work as well as you did imagined, at least compared to EdgeCast or MaxCDN. Although that is quite long ago.

Would love to see any recent input.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by toomuchtodo on Wednesday March 05, 2014 @12:22AM (#7345457)
There is no such thing as an open source CDN; there is a free CDN called Coral Cache (http://www.coralcdn.org/). CDNs cost money because you're dragging content from an origin to edge locations all across the world, and keeping it hot for client requests.

You could simply serve the content out of nginx, but you wouldn't see the performance benefits of keeping your content closest to the end user.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by joshmn on Tuesday March 04, 2014 @09:42PM (#7344899)
> The bias towards AWS for this type of application is ridiculous and a big waste of money.

They could get an even better deal by just going through a dedicated server provider (or even better, colocating).

There's little advantage with choosing DO versus going with a dedicated server provider (and again, colocating). I guess the advantage would be the control panel that they wouldn't use, having a few one-click stacks that they won't use, stuff like that.

If someone can afford a $7,000 AWS bill they can afford to put some money towards hardware and an onApp license if they want "cloudy" stuff. To colocate their hardware it would probably run them anywhere from $400-$800 a month depending on where they go. Their total bill would be decreased by $5500 a month. The upfront investment of the hardware wouldn't be more than $12,000 either. LOE? Probably two weeks with a competent sysadmin.

Yes you can have issues with your hardware and stuff and then you have to take care of that, but if you're good with your DC, they're great to you.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by vacri on Wednesday March 05, 2014 @04:00AM (#7346058)
at $600/month you've only saved them $1500/month (the hosting portion is only $2.1k), and now they also have physical hardware to manage, requiring a broader skillset from the volunteers, plus someone having to be in physical proximity for 'on-call' issues.

I don't know what datacentres tend to charge for data transfer, but as that's the largest item on the bill, it's the more salient point.

Also, just because it's not on the bill doesn't mean they're not using other AWS services; there are several free ones.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ceejayoz on Tuesday March 04, 2014 @10:03PM (#7344988)
> To colocate their hardware it would probably run them anywhere from $400-$800 a month depending on where they go.

For one datacenter, but CloudFront gets you 40+.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by gaadd33 on Tuesday March 04, 2014 @10:05PM (#7344996)
Bandwidth is by far their biggest cost, colocation/dedicated hosting would save a substantial amount but you are still going to be looking at something in the ballpark of $1,000/mo for 1Gbps. (Unless Cogent has slashed prices even further)
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by zimbatm on Tuesday March 04, 2014 @09:57PM (#7344965)
Digital Ocean doesn't provide a CDN. EC2 only account for 1.4k in the bill so I don't see how you would spare $5000.-
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by TylerE on Tuesday March 04, 2014 @10:21PM (#7345061)
Bandwidth on DO is (worst case), $20.48/TB, so their 17TB usage would cost $348/month. A far cry from $5,000.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ukd1 on Tuesday March 04, 2014 @10:38PM (#7345124)
You miss the point; bandwidth from a single source != CDN. Hosting without a CDN would be extremely slow for people that were far (hops) away from DO. CDN's solve this, and other issues.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by TylerE on Tuesday March 04, 2014 @10:55PM (#7345169)
How much does that really matter? Even going to the otherside of the world is only 200ms or so, and the time taken to run rubygems is hardly a factor in just about any workflow I can imagine.

Think of it another way - what would be more valuble - RubyGems hosted on CDN, or RubyGems on DO and give a couple of grants for talent hackers to work on their gems fulltime for a few months. (ala GSoC)

Even if you ARE concerned about latency, have one download server in the US (E.g. DO), one in Europe (e.g. Hetzner) and one in SE Asia (Not sure who's cheap and good-ish there), and you'd still be a 1/4 the cost of AWS bandwidth or less.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by evanphx on Tuesday March 04, 2014 @11:23PM (#7345263)
Person who pays the bill here. Latency does matter and we're paying an additional $1.5k new in Feb to improve european latency. RubyCentral can afford to spend the money to improve latency issues, so we do!
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ksec on Wednesday March 05, 2014 @12:05AM (#7345417)
Really, why would that be the case? I mean the difference in Latency.

A self set up Linode CDN with all six location Tokyo, JP London, UK Newark, NJ Atlanta, GA Dallas, TX Fremont, CA

Would have provided 48TB of Pooled bandwidth at a very decent speed and cost around $480. Linode's Network are great, much better then DO. I am not sure if it match CloudFront, which isn't exactly the fastest CDN anyway.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by sitkack on Wednesday March 05, 2014 @12:50AM (#7345544)
Great info. Did you use rsync and push data out in a tree? How did you handle DNS? Did you do 302 requests to closer servers (by some metric) ?
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by driverdan on Wednesday March 05, 2014 @12:38AM (#7345501)
Having volunteers spend countless hours to setup and maintain their own CDN is a preposterous idea.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by ksec on Wednesday March 05, 2014 @02:08AM (#7345776)
Here

http://psyphi.net/blog/2013/12/content-delivery-network-cdn-...

I have 10 fingers, so that is definitely not "countless" hours of work. And No, Maintenance are minimal or non existent. You could even get smaller VPS behind each node balancer as HA. Since Linode VPS ( Unlike DO ) are deployed on physically different hardware.

While i say it is fair enough to use AWS because money doesn't matter, i thought there are definitely some better alternative for the same price( if you really cared about latency ) or cheaper options.

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by NKCSS on Wednesday March 05, 2014 @03:54AM (#7346044)
Loved the blog posting :)
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by TylerE on Tuesday March 04, 2014 @11:31PM (#7345285)
Fair enough!
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by _delirium on Wednesday March 05, 2014 @04:08AM (#7346083)
That's been my impression so far with repositories where you configure a mirror explicitly, like Debian or CPAN. I used to be diligent about doing "the right thing" and switching out the default (usually something in the U.S.) for a Danish or nearby mirror. But I've stopped caring much because it doesn't really seem to make any perceptible difference. If I remember to, I still will switch it just so I don't unnecessarily waste intercontinental infrastructure, but it doesn't make much difference to my own experience.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by jdubs on Tuesday March 04, 2014 @09:38PM (#7344885)
It's often difficult to migrate providers when the application is complex or the owners may see the value in the provider. They're falling right into the hands of most cloud providers evil plans, they make it cheap to get started but as time moves on, it becomes more difficult to migrate away.
Re:Rubygems.org AWS bill for Feb 2014 [pdf] by vertis on Tuesday March 04, 2014 @10:11PM (#7345014)
We moved the stack to Amazon in about 60 hrs last year (gems were already on S3). Given that time involved writing a lot of chef recipes, I'd say if pushed we could move out again in an even shorter period of time.

Everything needed to build the rubygems.org stack can be found at https://github.com/rubygems/rubygems-aws

Re:Rubygems.org AWS bill for Feb 2014 [pdf] by xs_kid on Wednesday March 05, 2014 @12:00AM (#7345382)
I guess host it in AWS is a benefit for the integration with other services hosted in Amazon like TravisCI (the most popular CI for open-source Ruby projects) and Heroku (the most popular hosting for Ruby projects)