And on bandwidth their markup tends towards infinity - same on Azure and GCP.

As a example, 20TB of egress bandwidth will cost you around $1,750 on Azure, almost $90/TB! Your typical server/VPS provider gives you that kind of bandwidth allowance for free, because it costs them close to nothing.

I have a feeling almost all complaints about cloud costs would disappear forever if they'd only stop gouging customers on bandwidth.

The likes of Hetzner and OVH are starting to offer more cloudy features, like load balancers and block storage. My hope is that eventually they become viable alternatives to the big 3, and they are finally forced to reduce their ludicrous bandwidth prices.

The term 'roach motel' comes to mind when I see their network traffic rates.

"Come on in; there's no commitment. Pay as you go, and did I mention our ingress rates are free! Bring us your data and give the cloud a try. It's the future, you know..."

"Oh you're leaving and want your data back now?"

>20% minimum and typically much higher if you know how to negotiate when purchasing your on-prem gear.

A 20% premium on a $10k/mo server bill is an incredibly small premium to pay versus a single FTE.

I have yet to move a single customer to Amazon that fired the FTE who was managing on-prem infrastructure and chalked it up to a cost-savings move. . The idea of cutting headcount is a fallacy. In small shops the guy that was managing on-prem infrastructure is now doing that (because it turns out you still need switches and phones and routers and firewalls even if you host things in AWS) as well as managing the AWS infrastructure. In large shops you're typically replacing "cheap" headcount (the guy who racked servers) with someone significantly more expensive (a team of cloud architects).

We're a small company who avoided the hire. And we don't bother with a firewall at our office -- the only things on the office network are laptops, phones, printers, and sonos.

Basically, if you model senior eng time as $5k/week -- not even counting the opportunity cost -- you'll understand why AWS.

While I definitely think AWS is very expensive for the servers, they are not overall expensive. Again, setting up CloudWatch in an afternoon or dumping tons of heterogenous logs into S3 and querying them with Athena or spinning up full environment stacks (front end pool, back end pool, load balancers, databases, vpcs, vpns, etc) is an enormous cost savings vs finding, installing, configuring, and maintaining batches of software to do the same.

edit: pay an eng $180k. Fully loaded, you're looking at $220k plus. Add in either recruiting time or a recruiting fee that probably runs us $2k/mo amortized over the first year on the low end. Add in the fact that the first 6 weeks weren't super productive, and also absorbed tons of time from other SE to get this person up to speed. Add in PM time bringing this person up to speed on our product.

Yup, $5k/week.

Or, you know, you could have brought in a consultant part time. If you're small enough that you can handle an AWS setup without someone full time, you could run on a managed hosting setup for less including an external consultant charging ridiculous hourly rates.

Clients on AWS were always my best ones, because they were used to paying so much over the odds I could set my rates much higher.

Yeah, I wanted an extra person on our desperately understaffed team, so I proposed I cut about a million bucks of spending year-on-year from AWS.

They didn't want to increase the head count. The AWS spend was a different silo/team/contract.

Corporate accounting is insane.

It almost smelled like "well, we negotiated this great AWS discount, so it doesn't bother us that we are spending an extra million. We're getting it at a DISCOUNT!"

Is there some MBA trend where every headcount is some ticking time bomb of liability that is 10x worse than their yearly salary?

You spend 150k on servers, they don't improve over a year.

You spend 150k on a decent employee, not even rockstar ninja 100xer, and they will deliver useful, although sometimes hidden, productivity and optimization.

On top of that add the opportunity cost of spending all that time managing/setting on prem infrastructure Vs solving business problems....there’s a good reason AWS and the likes have millions of customers today.

It's such a meme on this site to compare costs of AWS to hardware costs as though the hardware operates itself. I'm sure there are many cases in which onprem actually is a better value proposition than AWS, and I would find it much more interesting if we talked about when to go with AWS vs onprem, but instead we try to make these silly comparisons about the upfront cost of a server blade vs an EC2 instance.

I've done operations stuff for ~25 years now. I've used AWS since it launched. I've done devops consulting for years.

I've yet to see an AWS deployment be competitive on cost with onprem or managed hosting. In fact, I used to make good money helping people cut costs by moving off AWS.

That includes paying more for devops, because AWS is incredibly complicated, and most people struggle to get it set up well.

There are valid reasons to pick AWS. Cost is not one of them.

Cool, this is the stuff that interests me. Comparing the nominal cost of hardware with the nominal AWS costs is boring.

Why? If the primary benefit you're getting from the cloud is VMs, it's valid to compare to hardware. The overhead of bare metal on top of cloud VMs is basically knowing how to handle RAID and set up load balancing.

VMs are certainly not the primary benefit of the cloud; cloud providers offer many services in addition to VM orchestration, but that hardly matters because your own example is illustrative of your error:

> The overhead of bare metal on top of cloud VMs is basically knowing how to handle RAID and set up load balancing.

This implies human beings to handle raid and set up load balancing, which suggests that you need to compare the cost of cloud providers with the cost of hardware and the cost of those engineering resources.

In addition to RAID and load balancing, most organizations/applications also need networking (good luck balancing load without a network), databases (including backup management), TLS, DNS, access management, etc, etc. All of this takes humans to build and operate. AWS services do a lot of this for you. In the on-prem world, you have to build (or buy/integrate) this yourself, but that's not free so you have to account for that cost in your comparison.

You can still make the argument that on-prem is a better value proposition when accounting for the total cost of ownership, but that's a different argument than those which ignore engineering costs altogether.

Yes, you need to know how to use Linux. And as the article astutely points out, there is an undiscussed problem with UNIX skills loss across the industry. AWS provides GUIs and Linux doesn't - accepted.

However for the many, many people who do already have those skills, and for the many pieces of software that are basically web servers + databases, the overhead of bare metal vs cloud VMs or services is basically a bit of Linux sysadmin work. People are acting like you have to hire a full time wizard to even consider using anything other than high cost AWS services but that is wrong: AWS requires learning too, people who understand UNIX are plentiful even if apparently not as plentiful as they once were, and a well run setup will not require constant sysadmin work. A well rounded developer will often be able to do the ops work as well.

Also "bare metal" doesn't mean running your own datacenter. You can buy hardware, send it to a colo and they'll rack it and run the network for you. That's why I say it's mostly a matter of understanding RAID: when the hardware fails (almost always disk or less commonly, RAM units), you file a ticket, blink the HDD LED and some remote hands go and swap out the part for you. Those "hands as a service" come cheap compared to AWS.

A large part of the problem is that most developers have no experience with this, and think server hardware is as much a pain as home PC hardware, which a large portion of younger developers have little experience with too.

They've not seen servers with IPMI tied into PXE boot and a TFTP setup. They don't realise once that server is racked up, we can log in to an IPMI console, power it on remotely, have it boot straight into a bootstrap script, remotely install an OS like CoreOS/Flatcar without manual intervention, have orchestration scripts install the necessary basic services on it without manual intervention, and boom, it's part of your own "private cloud" ready to deploy containers to that may well not require maintenance "below" the container layer for years.

And they don't realise you can contract someone to do this for you for low enough day rates that it'll take you a truly massive setup before you're paying the equivalent of a full time engineer.

100% agreed. I venture that it's a side effect of our industry relying almost entirely on universities and self-learning to propagate knowledge, instead of trade schools. Academics don't want to teach "messy" real world skills like how to actually build a distributed computer with your own hands: there's a very narrow range of skills considered acceptable to teach in the academic world and sysadmin isn't one of them.

Meanwhile we've had 20 years of media and culture telling people that those who get rich are the software guys, never the infrastructure guys, and that the easiest way to get into the industry is by learning web design.

Also, I suspect the industry is exhausting the pool of talent that learned computing on UNIX workstations at university in the 80s and early-mid 90s. The people who lived and breathed servers and UNIX are retiring out of the workforce, and not really being replaced. AWS is stepping in the fill the breach by selling sysadmins-as-a-service, basically, with a thin veneer of GUI programming slapped on top, and some proper documentation writers. Selling pickaxes and shovels to the gold miners - classic.

Finally, I think a lot of firms have developed a "no contractors" policy. I've never worked at a company that allowed contractors and I'm about mid-way through my own career, a bit less. At some point it has become received CEO wisdom that they're building a committed team of true believers, so "mercenaries" aren't welcome. It leads to the kind of thinking you see in this thread where "I don't have the skills, therefore I must hire someone full time, and they must earn as much as me, therefore it's too expensive" ends up leading to "and therefore I must outsource to a firm that 'contracts' for an hourly rate billed via credit card".

The contractor has a problem similar to employee turnover. Contractors make sense when there is a throw-away project (won't need long-term maintenance) that doesn't embody any proprietary knowledge or skill. As soon as a project could expose a trade secret or require on-going support, the contractor becomes a big risk.

If you don't have an FTE in charge of servers, who's managing AWS? If you leave it up to the individual developers you end up with an overcosted mess.

But again, is that comparable? Seems like apples versus oranges.

What’s the connectivity into your 7+ year old asset cost? What about the space, power and cooling to provide it a home?

What happens when your environment gets attacked by a volumetric DDOS, is that 100Mbps circuit going to get overwhelmed?

I’m not arguing AWS is cheaper, simply that there are lots of things around the physical server which also have costs and complexities.

>What’s the connectivity into your 7+ year old asset cost? What about the space, power and cooling to provide it a home?

10Gbe ethernet was standard in 2013/2014. Space power and cooling is company dependent. If they're tiny, a closet in the building or a colocation facility, if they're larger, a room. If they're huge, a datacenter.

>What happens when your environment gets attacked by a volumetric DDOS, is that 100Mbps circuit going to get overwhelmed?

I can't recall the last time a customer got hit by a "volumetric DDoS". End-user facing systems sit behind cloudflare. Internal apps are... internal. Who cares if the WAN connection is getting hit by DDoS when your users are on campus? In post-covid world, they've got multiple tunnels with SD-WAN.

That's ignoring that fact that Centurylink or Comcast or Telia are pretty good at mitigating DDoS in 2020. Just because they don't post about the attacks they endure on a blog doesn't mean they're inept.

https://business.comcast.com/help-and-support/ethernet/comca...

https://assets.centurylink.com/is/content/centurylink/centur...

>I’m not arguing AWS is cheaper, simply that there are lots of things around the physical server which also have costs and complexities.

Sure, and they're all complexities which we solved 30+ years ago.

> Sure, and they're all complexities which we solved 30+ years ago.

The argument isn't that they are solved or not, but the cost of implementing and maintaining them vs AWS.

I never say a whop running on AWS and not having a person managing it. Said person always has enough skill to replicate same setup on a dedicated hardware/VPS. I don't say such shops does not exists, it just I never saw them.

To me AWS killer features are RDS and ability to provision 100 servers in a few minutes. Most of the time scaling is not needed and RDS alone cannot justify the cost of AWS.

For a lot of companies, 20% is an acceptable markup to pay, in return for not having to deal with hardware failures, delays due to ordering new hardware, not always having instant failovers, over-purchasing for capacity/scale, the extra human dev ops costs, etc. If you're well-positioned to handle that, then great, AWS may not be cost-effective for you.

>And if you happen to be a shop that sweats assets for 7+ years that number starts being measured in the hundreds of percentage points more expensive.

If you include the growth in electric efficiency over time, it can be bad to run old servers after a certain point.