At Reddit's scale they would be able to negotiate directly with AWS for lower rates. I'm sure they already do so with servers.

[ deleted ] -- I removed it because I got tired of trying to explain the nuance.

That is absolutely false. Plenty of large companies are able to negotiate their rates on AWS. You just have to be huge.

Do you realize that the person you're responding to used to run Reddit's servers?

Is that why the comment was deleted?

I didn't say he was right. And I also have no idea why his comment was deleted.

[ deleted ]

Hardly huge. When I was at a web agency with about 70 people on staff here in Australia, we were an AWS partner with negotiated rates, and honestly we weren't doing massive clients really (aside from some government stuff now and then).

How much was the monthly bill ($) ?

[ deleted ]

It has happened, though I'm not sure if it's common or not. I worked for a big hadoop user (5 years ago: 6k boxes/over 50k cores/30 pb) that aws really wanted on their platform. They offered huge discounts to get us as a marquee customer, much better than their public pricing. Unfortunately, they were still ludicrously expensive compared to O&O but we were good at running huge automated clusters with 1.5 ops.

You don't have to be huge, just not tiny.

Depends what. We pay significantly lower than label price for Cloudfront prices. EC2? Never seen a reduction in that

TIL. After reading about Spotify negotiating with Google Compute Cloud I just thought it was common place for such orgs to negotiate vs everyone else that will rarely reach that scale.

According to Reddit's source code they're using a provider called Imgix - https://www.imgix.com/ - who I assume use S3.

Came across an incredible photo series of their dataserver. http://photos.imgix.com/building-a-graphics-card-for-the-int...

> Apple’s operating system offers numerous advantages over other platforms when it comes to image processing, and it is also favored by many designers for the same reasons. imgix has written tools which allow us to leverage these strengths with our service, using OS X for image operations where it provides superior quality and performance.

What advantages? If you're running these as headless servers, then what advantages does OSX possibly provide over cheaper, more powerful Linux boxes using e.g. ImageMagick (ok, ok, not having massive remote code execution vulnerabilities is one of them).

Same thing jumped out at me--having racks of mac mini's seems like a waste

Does ImageMagick have access to GPU hardware acceleration?

Of course: http://www.imagemagick.org/script/opencl.php

Crazy.

Who in turn look like they're using fastly for their cdn judging by them sharing the exact same pops.

(Guessing they use the fastly private cdn or have some sort of deal with them considering the imgix branded cages on their blog).

https://docs.imgix.com/guides/cdn-guidelines

https://www.fastly.com/network-map

They're using reduced redundancy which should save them a pretty penny.

What are some better alternatives to s3?

Colocation hosting your own servers around the world... you know, the original way.

s3 is awesome. Keep using it but put your own boxes in a colo with a direct connect to aws (not all colos offer this unfortunately). Your boxes can then g3 20+tb of ssd cache in front of your raw pb+ datasets in s3.

https://aws.amazon.com/directconnect/pricing/

Even cheaper and easier is just to rent dedicated servers or VPSes.

One of the earlier Imgur style hosts, ImageVenue, did this very successfully (they were extremely large in terms of traffic at their peak and for the time, until better options came along).

The founder, Vlad, used to rent out cheap dedicated boxes with about 500gb of bandwidth per box for $60 to $80 per month. He'd cycle upload machines - eg 123.imagevenue.com - and fill up the drives on them until no more space was available; when bandwidth was exhausted, that was that for the month. Typically image file downloads would decline on any given upload over time, so machine resources would free back up in terms of bandwidth and deleting images that weren't viewed after N amount of time.

For image-heavy use I found its just cheaper to build your own box and cohost instead of renting.

How do you figure that? You can rent the exact same server as a dedicated server. The difference is, you have no capital outlay up front, you don't need to ship and rack it or replace broken parts and there is only a month-to-month commitment.

The boxes with lots of disk and memory are much more expensive when rented vs buying. SSDs are super expensive to rent, for example.

I made a Skylake Xeon system with all enterprise components for $2k. You can't even rent Skylake Xeons anywhere right now.

I've never had problems ordering custom hardware from my dedicated server providers. At worst, all I've had to do is pay a one-time fee for a particular component. SSDs have never been a problem. Neither has any Intel CPU.

It of course depends on your provider.

Rent vs buy depends a lot on the circumstances. Maybe if your requirements don't change and you don't upgrade your server hardware it's cheaper in the long run to buy. However, when I've kept a server for more than a year, I've always been able to negotiate a discount.

You didn't post your config, so I can't tell how exotic your requirements are.

Is that really cheaper than S3, though?

Even at Amazon's cheapest posted bandwidth tier you're paying $50+ per terabyte for bandwidth.

Any dedicated hosting company (OVH, Hetzner, etc...) will cost less than 3% of that (and that's on the high end).

Bandwidth is cheap, but AWS marks it up roughly 30x-50x what you'd pay in a dedicated environment.

Then why isn't anyone running an AWS-like service focused for bandwidth-intensive applications hosted on "OVH, Hetzner, etc..."?

Many people do. S3 is convenient though. It handles durability, replication, uptime, etc... and even with the insane markup on bandwidth many startups / companies can afford a $10k/mo bandwidth bill because it's cheaper than hiring some devops and a team to build out the appropriate infrastructure.

That's what I meant with "really cheaper". Sure, bandwidth costs are lower with Hetzner, but that's not everything.

It depends on how you define AWS-like services. Lots of VPS providers cater to bandwidth-intensive applications and are hosted on cheap dedicated servers.

But you don't have to store and serve all the content from Amazon or your own platform. You can easily set up aggressive caching, in a network you control, backed by S3.

And you pay premium for premium bandwidth. You can't just assume that all your users will have a good experience if you buy hardware and bandwidth from the cheapest providers around.

Amazon bandwidth isn't particularly premium. It's just expensive.

Way cheaper. Amazon is a middleman for hardware. Even at smallest scales the cost of buying a server pays off within a year. In exchange you get convenience and the ability to scale up on demand without buying more hardware.

I'm sure buying the server pays off, but what about running and administering it?

What's the minimum commercially you'd spend to get cross-continent redundancy on hardware you manage in-house. Just the accounting cost of paying wages for technicians in two countries probably means you need to be a quite large company.

There are middle grounds, managed co-location say, but then you're not likely to save much over using AWS are you?

Even managed co-location is cheaper than AWS when you factor in the egress bandwidth charges at scale.

Another way to do it is to use dedicated servers. The datacenter will take care of repairs and remote hands functions and all you need is to manage the server from the OS level upwards. No need to have technicians in multiple countries, everything can be done remotely.

Yep this is what we do. It works well.

You still have to manage/ administor the aws instance. It only really saves you from the overhead involved with hardware failures.

Yes, way, way cheaper. Amazon egress bandwidth is 10-100x more expensive than IP transit.

Depends on what you compare it you. Amazon sells you not only the hardware, but also the management services.

Massively cheaper at scale. Do the basic math.

can you explain the math? I honestly don't have a working knoweldge of the inputs, but let's say i wanted the equivalent of ec2? How would I go about setting it up and the costs? If I wanted to serve a few million english speaking people my web application, how coud I calculate the price? Do I literally buy my own hardware, or I rent hardware at a dedicated rate? These are real questions btw, not rhetoric, it's hard to find a solid blog post or comparison of setting this up and the long term cost v convenience benefits of scaling this because almost everyone uses cloud providers, and by cloud providers I mean digital ocean, AWS, azure, rackspace or now google. Heroku (last I checked) sits on AWS as well and maybe even ngineyard does. actually pretty curious now if a few million people would be worth running your own gear, and if running that gear was lease v buy

The cost difference depends on how much your administration costs go up because you need to do things yourself, rather than through the AWS API. That's why it's hard to calculate.

Not sure if this is just for your private photos, but it's cheaper: https://www.backblaze.com/b2/cloud-storage.html

Use S3 but put a cache in front of it, at the very least.

Reddit use Cloudflare on both reddit.com and redditstatic.com

> Can't imagine that AWS bill...

That's not how things work over here.

We can and do imagine. We even calculate estimate values so we don't have to stay in the dark and reason without a solid basis.

I'd encourage you to find some traffic numbers provided by imgur and some S3 pricing to do a rough estimate. Maybe someone more knowledgable will pop in and improve on that. When we argue everyone involved shall be wiser after. That's how it works around here.