Traffic's a luxury tax (along with RAM) that cloud providers assume that big companies can afford to pay if they're getting that much traffic.

Outside of the Cloud Providers Traffic is dirt Cheap, Hetzner includes 30TB traffic inclusive in their dedicated server i7 Quad-Core Skylake 64GB DDR4 RAM, 2x250 GB SATA 6 Gb/s SSD for 39 euro /month:

https://www.hetzner.de/us/hosting/produkte_rootserver/ex41ss...

If you don't want to be shaped after you exceed 30TB, Hetzner charges €1.17 per additional TB, so 700TB would come to €783.90 total.

Whereas ScaleWay include unlimited traffic in their bare metal servers starting from 12 euro /month:

https://blog.scaleway.com/2016/03/08/c2-insanely-affordable-...

It sounds kind of inefficient though since different business types have extremely different bandwidth needs. So it's going to tax business by sector rather than by their ability to sustain it.

How many people share that Hetzner server for 39euro/month?

It's fascinating for me to see again and again people somehow accustomed to cloud pricing hear about bare metal hosting offerings and not to believe the prices could be that low. BTW this applies not only to traffic, but also processing power and storage.

If you are looking for a bargain with little commitment, you might want to take a look at Hetzner's auction.

https://robot.your-server.de/order/market/sortcol/ram/sortty...

The gotcha is the allowable bandwidth for their auction machines are lower than their normally priced servers.

https://www.hetzner.de/us/hosting/produktmatrix/rootserver

I got lucky and found a 32GB machine with 4 Samsung SSDs in their auction and it has served me well for testing. I write about it my blog below:

http://gitsense.github.io/blog/benchmarking-march-14-2016.ht...

I think I'm paying about 68 euros a month for it. The Canadian dollar has taken a beating, so it's not as good of a deal as it use to be, but it's still a good deal none the less.

It's a dedicated bare metal machine for you. The tradeoff with Hetzner is, that it's not expensive Server hardware, so you will encounter hardware problems more often compared to a Dell or HP server.

You just have to build high-availability into your software. I'm using six Hetzner servers for over 1.5 years now and the only problem I had was one disk failure - support needed 10 minutes to swap it. I can highly recommend them! I pay them ~200 euros per month for what would cost me 2.000+ dollars on aws...

I'm curious about this statement : is that as opposed to cloud apps ? Wouldn't you need to build high-availability into your apps whether they're running in the cloud or on dedicated ?

If you mean that you can have a load balancer in front of it managed by Amazon, that's true for dedicated as well (Akamai, CDNetworks, Limelight, even Leaseweb). Managed databases are available from most providers (usually without an API, but you can find them with an API as well).

Failures will happen no matter what. About the only difference I think you'll see is that most of their servers don't use ECC memory, so you're technically more likely to hit a problem there.

I've had one server with them for about 3 years, and another for 2 years, and haven't run into a hardware issue yet. Obviously a hardware issue could happen at any time, so anything I can't live with being offline until I can restore from a backup is configured with redundancies, including a Digital Ocean VPS just in case the datacenter my servers are in goes offline.

From my monitoring, however, I tend to see a short network blip about every other month, but it's less than a minute at a time. All other outages I've had were my own fault.

Hetzner also has options for Xeons and Dell PowerEdge servers for a bit more a month, but I've also had great experience with their best value hosting servers, ran a site on it for a couple of years without running into any h/w issues before moving to AWS due to its easy managed RDS, S3, SES services. But if I'd just needed a single dedicated server with great specs I'd use Hetzner in a heartbeat.

I picked up one of their new ex41-ssd machines and I'm actually kind of nervous about it. I've bought a few auction machines and they all came with Samsung SSDs, but these new EX line machines are using Crucible, hence the low price. I have zero confidence in Crucible and I'm not sure if I will buy anymore EX machines, unless somebody tells me Crucible has a different reputation now.

Ok, cool. The price just seemed low.

Hetzner is towards the vey low end of pricing (downside: latency if your users aren't in Europe), but dedicated servers from most providers end up far cheaper than AWS or GCE.

None.

Actually, one. :)

Its a full root server. In fact you can get an older root i7 for 20-25 euro

I don't get it. Google says they're going after the big fish in the industry by claiming they have amazing pricing. The servers look good, I'm ready to jump on board.

$120-$230 for first TB of egress bandwidth depending on where it goes. No thanks, I can get 2 TB for < $20 elsewhere.

These bandwidth costs leave small businesses, and individuals like myself, staying with the smaller competition. I suppose their reasoning is they can chase after that single $400-600 million contract. One major client like that is worth as much as ten million of us little guys paying $50 each. The big cloud providers exist to to serve gigantic enterprises. The rest of us are a drop in bucket and not worth the effort.

When pricing a value-add you want to price it linearly, with a volume discount, but such that after the volume discount the line is still steeper than the base cost curve. That way growing customers feel like they are getting a deal vs small fish, and are incentivized to use as much as they need, but you still drive your margins towards what the market will bear, provided your volume is growing. That curve will eventually squeeze out some of your biggest customers, but you can avoid this by cutting deals for them, e.g. Google with Apple.

Traffic is not important for every use case. If you run a store for physical items, how much traffic are you going to use? This is probably going to be less then 5% of your AWS bill, so you don't worry too much about it. If you host heavy images, big JS files (which you shouldn't do anyways) or offer downloads, you should probably use a CDN anyways. For big downloads, latency is not really that important as long as you get proper download speeds, so the CDN is going to be a lot cheaper.

Nor everybody wants to run the next Netflix or Dropbox in terms of bandwidth consumption. Even if you did, keep in mind that Netflix does not host the videos in the cloud.

Cloud, especially AWS is 10+ times more expensive that hosting same stuff on DO, Vultr or Bare metal. And you still need administrators, EC2 are just VPSes like any other service from other companies.

They are pricing themselves out of the market for traffic-intensive small-fish operations that way though.

Why do you think they want any of that action? I think their pricing conclusively demonstrates that they don't. Some of those customers are waaaay more trouble than they are worth. Also Google and AWS have "premium" bandwidth - massive redundancy and lots of peering relationships.

> traffic-intensive small-fish operations

Do you have any examples? It seems like it's always been a grow-and-become-profitable-or-die-fast niche.

I run a few websites with video content which leads to 50TB+ per month. The business is profitable, but clearly I would not waste my money on expensive bandwidth.

Google clearly isn't trying to be a porn CDN

Which is funny because through YouTube they have to have the cheapest raw bandwidth in the world.

They need two traffic prices.... Fast low latency web traffic for the current 10 cents per GB. Slower more laggy CDN type bandwidth for like 10 cents per TB.

Don't Google have a CDN service?

Edit: yes. (https://cloud.google.com/compute/docs/load-balancing/http/cd...) But it's more of a CloudFlare competitor—a distributed caching reverse-proxy with a 4MB object cachability limit. Costs $0.008/GB, which is cheap compared to a real CDN, but expensive compared to CloudFlare's "free."

You miss out something there, I think. The $0.008/GB is for the load balancing. On top of that, you still pay for network egress depending on whether it is internally in GCE or to the internet. Those rates are from $0.20 to $0.08 depending on location. (EDIT: For traffic to the public internet)

And those rates are still in crazy territory compared to most alternatives other than Azure and AWS which have equally messed up bandwidth pricing last I checked.

I build caching solutions for customers that want to store their data in S3 or Google Cloud Storage, because the bandwidth prices at the big cloud providers are so out of whack that as soon as someone uses lots of egress (few TB a month or more), you can often cut your bandwidth costs by 80%+ or more by getting some dedicated cache servers to put in between your users and your cloud storage. That is after the rental and management costs for those cache servers are included.

(the reason for this rather than building storage solutions is that if the above fails you don't lose data. If you trust your abilities or service provider, building a multi-location storage setup with 3+ times redundancy that beats S3 etc. on cost by a large margin is fairly straight forward... But it's often easier to sleep at night if you have other people do the risky stuff..)

Funny guess, but wrong.

If you have time, I'd recommend purchasing dedicated servers in multiple geographical areas and setting up a custom CDN. It's much cheaper, however much less reliable and much more time intensive to manage and diagnose.

Which DCs would you recommend?

you can try finding good deals in areas you are interested (be certain to ask for "test" IPs, then look up their connectivity via _multiple_ looking glasses) on www.oneprovider.com and then pair that with a robust DNS provider such as NSOne and you've got yourself a pretty decent, bespoke CDN. provided you already know how to do reverse caching proxies and all the other "magic" a CDN needs to work.

Wow, ~700 TB/mo? That does sound like a lot.

What kind of site would serve that volume of traffic and not have 56k for operating expenses? I mean, I can think of a few examples like Wikipedia maybe, since they are non-commercial and such, but for a commercial business? Maybe 4chan moves that much without a lot of revenue I would think, or maybe... imgur? but not really sure, I mean, it would seem like they could get that amount easily via ads alone.

What was the use case here?

Also, I think that 56k for traffic alone kind of depends on context. I mean, how much does Netflix pay for serving their volume of traffic?

What I'm saying is, isn't 700 TB a month something that would probably be very expensive no matter the context? Just storing 700TB would cost a lot, no?

I'm really curious about your use case here.

Image hosting community site - notably without shady popup/layer/scam ads, which probably was the reason for the relatively small income. For a two person team that only worked part time on it, it still made good money.

The total dataset was just about 3TB, so storing it was not an issue.

I see.

It does make sense. Thanks for satisfying my curiosity :)

700TB/mo is about 2Gbps - on the open market that should be under $1000/mo. Netflix's total cost is probably below $0.25/Mbps. $56,000/mo would get you over 100Gbps of committed capacity from any major provider (or a mix).

> 700TB/mo is about 2Gbps - on the open market that should be under $1000/mo

Is that a fixed-cost sustained pipe though? I was under the impression that (at least at the backbone level) those contracts got more costly the closer to full that your pipe was.

Yes, $1000/mo would get you a 2Gbps commit on a 10Gbps pipe. If you used over 2Gbps (95%ile) for the month, you would pay probably $0.60/Mbps for that excess ($0.10 over the commit price). Some providers dont charge more than the commit price for overage traffic.

Interesting data. Thanks!

It's obvious that I had no idea about costs. I honestly thought it would be much more expensive.

Why is it "not have 56k for operating expenses"? Something that can be had for $2k is not something a healthy business spends $56k on. You should be able to find a better use for those $650k that year.

Well, mainly because I had no idea about the actual costs. I kinda spoke too soon.

I just thought that moving that much data would cost some serious money. Apparently that's not really moving "that much data".

Wrong assumptions on my part :/

The engineer time to reimplement the other AWS services you're using may be substantially more than the $54k difference in bandwidth costs.

Depends on what they're using AWS for. OP seems to be running a simple and straightforward setup that just happens to use a lot of bandwidth. It doesn't take five full-time engineers to maintain a handful of LEMP servers.

Nothing prevents you from mixing and matching there are actually AWS services you can't find cheaper equivalents to elsewhere. My experience is that those of my consulting clients that want to migrate off AWS rarely have any problems replacing it. The cost savings usually pay back any development costs and the overall migration effort in 2-3 months at most.

If there will be 2 such companies, for that 108k I can wrote services backend that will be comptabile with AWS, so after a year you can transparently switch to that system on your bare metal and sell it to have your services for free.

I still didn't make my own "clone", because I can't afford machines to start selling it.

Hmm very unlikely. Thats 5 full time people.

More like 2 or 3 if you include overheads. Depends on location, of course.

It's where they make their money. Like when a restaurant pushes the desserts on you - the desserts have the highest markup by far on the menu.

AWS has a lot of 'free' services, which still have to be paid for. Some of those free services are things that benefit both the client and AWS, but would be avoided by many if folks had to pay for them (like IAM credentialling)

1TB/mo is roughly a constant 3mbit/s. So an estimated 2.1gbit/s. I recently had a 1gbit line from he.net quoted at $500 in Seattle.

One popular high traffic site I know build their own CDN to serve the large majority of their data by renting dedicated machines in OVH, Hetzner, etc. I can not remember their actually datacenters for their own CDN but they were not CloudFront or Google Cloud Platform.

Supposedly this has saved them immense amounts of money.

If your servers are efficient enough (and this is not hard to do these days), it's easy to get bandwidth-limited on a per server basis, i.e. your server could handle more traffic, but you've maxed out the bandwidth available to that particular server.

If you can load balance at the client, then you can "talk" to any server at the edge and don't need a router or proxy, so the net result is that you are only paying for whatever bandwidth comes with your OVH (or whatever) boxes. Effectively, you're buying bandwidth and the computer/storage/power/rackspace/etc. that comes with that bandwidth is free.

And yeah, it's ridiculously cheaper than AWS or Google's Cloud Platform to do things this way.

> Can someone explain to me why traffic is still so damn expensive with every cloud provider?

Because The Cloud(tm) IS cheaper--when you start and don't have any real bandwidth or CPU usage.

Whereas, every colocation facility I have quoted wants you to commit to a minimum of $500 for some partial cabinet. So, The Cloud(tm) wins the contract and gets to bill in increasing amounts when usage finally goes up.

Finally, how many real system administrators still exist who can provision your systems, configure the network, and understand how to connect everything to the network without getting p0wn3d? If you don't have that person, you can't escape The Cloud(tm) even if you wanted to.

> Finally, how many real system administrators still exist

... a lot? Has there been some shortage of network/infrastructure people lately?

Well, considering how many small/startup shops expect the developers to also do IT chores, "the cloud" makes the most sense... spending time learning the insides of systems they don't care to truly maintain comes at a cost... time to do other things, or cost to pay someone else to do it.

In the end, the cloud makes sense in a lot of scenarios.

"The cloud" does not mean you don't need real system administrators. I see time and time again companies get bitten by this. Overall devops efforts to run this well on AWS or GCE in my experience tends to be higher than provisioning dedicated systems because you have so many artificial limits imposed on you by the providers that makes things harder.

E.g. your example: Understanding how to connect everything to the network without getting hacked is far easier when your private network is physically wired to a separate switch, and your public network is physically behind a firewall and there's no configuration mistake in the world you could do that would change that, so the problem-space to get basic levels of security is reduced to configuring the firewalls correctly.

Still plenty of room to shoot yourself in the foot, but in my experience far less so than having people configure their own networking on AWS.

As or pricing, yes, if you want to do colo, the initial costs are higher. But dedicated rented servers with monthly contracts are also typically far cheaper than AWS for anything that stays up for more than ~1/3 or so of the time (obviously depends on the hosting povider). If you regularly spin up lots of instances for a short period of time, you should use AWS. But the moment you stop spinning them down again, it's time to rent capacity somewhere else.

Perhaps it is like the gas stations that sell gas for $4.99/gal when others sell it for much less. It's only worth their while to sell it if they make a healthy margin so they only sell to people willing to pay that much.

Storage is also a lot more expensive from 'cloud' providers, people often forget to look at the performance and redundancy and simply look at 'per gb' costs.

Indeed. The IOPs numbers for the cheaper VMs are not so great.

You need IOPs? Suddenly you are paying for a premium instance type.

You want replication and/or geo-redudancy with that? Now we're talking $$$ :D

To clarify, we don't do that on Compute Engine. The number of IOPS you get is tied to the volume size for Persistent Disk. You choose between the two flavors (SSD and regular) and then size your disk. That does mean you have to buy more GiB than you "need" if you want to go faster, but PD is much cheaper than "bigger VM" in most cases.

Disclosure: I work on Compute Engine.

So lets say... how much would say a 2TB volume providing a consistent minimum of 100,000 random 4k write IOP/s that's available across multiple VMs at once and must be highly available at say 99.9% cost?

* Note: I went to use Compute Engine's cost calculator but it appeared the site was down / under heavy load?

Seems like the obvious cynical answer is that they do that to encourage you to use more of their services.

Their CDN interconnect lowers that pricing to ~$0.04/Gb (US).

That's still very expensive. Wholesale rates for bandwidth are a fraction of a penny per GB.

That is still about $13/Mbps, or 26x transit pricing.

This isn't wholesale, but at least it's half what they quoted. Additionally, you're only paying to update the assets and CDN fees.

How much do staff salaries and data center rentals add to the cost per server and per GB?

Why do you assume you need a staffed data center to get cheaper bandwidth?

Just buy dedicated servers or VPSes, no datacenters or staff needed. The hosting provider takes care of the servers, staff and the datacenter.

If I ever got to where my bandwidth fees were even a hundred a month for personal projects, I'd switch over at least part of it to a VPS...

As a business, I wouldn't do it until the cost of the bandwidth+hosting exceeded the cost of an extra, dedicated employee to manage the VPS server(s).

> As a business, I wouldn't do it until the cost of the bandwidth+hosting exceeded the cost of an extra, dedicated employee to manage the VPS server(s).

Why do you think you don't need that extra person to manage the instances in a cloud setup?

My experience is the reverse: It tends to take more man-hours per instance to manage a large cloud setup, because there are many more spinning wheels. The overall complexity is often vastly larger. In fact, I have clients I manage physical servers for where the time taken per server is on average still far lower than for cloud instances even including the 2+ hours lost on travel per visit to one of the data centres if someone has to physically go in (rather than rely on "remote hands").

This is before factoring in typically higher utilization rates for the dedicated hardware, because it's easier to customize it to get the right balance of RAM, CPU and IO for your workload. The result is usually fewer dedicated servers than you would have cloud instances.

If I'm using RDS or Azure-SQL, I'm not managing a database server... The list goes on, but when starting, you may only have one person or two working on actual development... features are important... actual customers and actual revenue may well be more important than scaling to millions of users.

Working is better than not working perfectly.

How do you figure you need extra, dedicated employees to manage VPSes compared to cloud VMs?

The hosting company takes care of the VPS servers, just like Amazon takes care of the AWS servers.

Once you need to scale, you need that expertise... if I can use RDS, Azure SQL, or a number of other options to manage database services, or other systems without dedicated staff, that buys time to keep the lights on while actual solutions and features are created... an MVP needs to work... And "wasting" a few hundred a month on hosted services while trying to get something working is better than having to spend that time becoming experts on infrastructure, databases, or any number of other systems.

I'm not saying don't optimize, but I am saying that you shouldn't switch infrastructures unless you are saving enough to cover additional talent.

That is a key question I have been pondering myself.

One theory of mine (perhaps uninformed; I'm not really a networking expert) is that because of the dynamically configurable nature of their systems, they need to use routers rather than relatively dumb and cheap switches at almost every level - in order to have flexible networking and still maintain isolation between customers.

This could get quite expensive if you have to pay Cisco/Juniper for this. If this is true Google will have quite an edge with their software defined networking here, I would guess.

No, they use whitebox switches and software defined networks to control. See https://www.youtube.com/watch?v=n4gOZrUwWmc [Edit: oops, fixed!]

That's Google. They have put their cost levels somewhat below Amazon's. Maybe they don't see the need to be 5-10x cheaper than the market leader in traffic costs even if they could...

> See https://www.youtube.com/watch?sns=tw&v=z7Cyv3cvIxY

Nike advert?

Lol, sharing multiple youtubes with multiple people at once. Fixed.

SDN is changing the model here, and Google is way ahead. In an enteprise, you can use VMWare to do a lot of the stuff you are blowing big bucks on for Cisco/Juniper on and use switches with higher density.

SDN is going to turn the cost structure on its head -- I wouldn't want to be a network guy now, easily 60% of tasks are getting vaporized in the datacenter.

As a network guy, it's a _great_ time to be an experienced network person. The only mature aspect of the ill-defined SDN sphere is OpenFlow and that will only get you so far. Try as they might, controllers like OpenDayLight and the various things that plug into Neutron/OpenStack aren't plug and play for those w/o significant network knowledge.

From my vantage point, it's going to be at least another five years before the cost structure really does turn over on it's head for folks below the hyperscale level

No doubt. Any change is great news for smart people. But the average joe churning out firewall changes and similar are screwed.

Google's really ahead on the networking front, and other cloud providers are following suite. Networking hardware is super cheap now. When you couple that hardware with open source software networking gets cheap.

Large networks like Level3, Cogent, Telia, etc all use big-iron routers (Cisco/Juniper) and will sell you traffic for under $1/Mbps.

Yep, and once you're at the multi-gigabit per second level, the price drops much lower than that pretty quickly.

Indeed - Internet Transit at scale (10 Gigabit+ ports) goes for around $0.63/Mbps at 95th percentile. [1] - for the above quoted 700 Terabytes/month, that works out to $1341/month, if it's evenly spread out on the lower 95th percent of the circuit at around 2.129 Gigabits/second.

[1] http://drpeering.net/white-papers/Internet-Transit-Pricing-H...

They (Level3, Cogent, Telia) don't have millions of ports though...

This is true, but I can't imagine Google/Amazon/Microsoft are using Cisco/Juniper routers at every level of their network.

Hosts play a big part of SDN in that they support the dvswitches along with guest VMs. Not everything is a Cisco/Juniper. Switching hardware is still common in TOR and egress.

VLAN's and Virtual Appliances in the same environment as the guest machines to facilitate routing should allow for scale without costing these virtualization providers too much.

$1/Mbps per what (unit time)?

Per month, usually (billed 95%ile). That price will decrease by quite a bit with more volume.

Thanks. How long has it been roughly $1/Mbps/Month ? Do you know of any sources with historical data?

Average was $0.63/mbit in 2015, and I personally haven't seen lower than $0.40/mbit.

http://drpeering.net/white-papers/Internet-Transit-Pricing-H...

That really depends on volume, location, and provider. For large volumes and cheaper providers (Cogent, HE.net, etc) it's been that way for 2-4 years or more. HE.net will now sell a full 10GbE port for $2600/mo, Cogent isnt too far behind. Sub $0.40/Mbit at >25Gbps volumes in major locations is doable.

pretty sure (at least) AWS builds their own network hardware. I remember reading something a while back that said they found it magnitudes less expensive than buying enterprise hardware, with better performance as they went about the affair as scientifically as you'd expect them to.

Old CoyotePoint routers were just a commodity x86 motherboard with an ancient SSD instead of spinning rust. Junipers use a duo of x86 (routing engine) and ASIC (packet forwarding engine). Cisco has supposedly moved from that architecture to an ARM and ASIC pairing.

The ASIC is just a hardware offload for known routes. Unknown routes, admin work, and Ping packets are handled by the x86/ARM CPU. It's not too different from offloading graphics work to the ASIC on your graphics card, or your mining to your Bitcoin ASIC.

> How's that justifiable?

What, morally?

Sounds like you should start your own cloud hosting service! I bet you could make a killing.