But I do think he's performing a useful service in highlighting just how much you're paying for not having to hire a devops or sysadmin person. 10x performance differentials have been replicated not just by him but in other benchmarks. I remember seeing a chart when I was still working at Google and GCP was being justified that showed a graph of hard disk prices vs. S3 rates - since inception (2007-2012 at this time), S3 rates had gone down by a factor of about 2-3x, but price/GB of physical hard disk space had gone down by ~100x. Amazon is hiding all of this improvement in the physical hardware behind vCPUs and opaque billing, and it becomes pure profit to them.

I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins. At that level you could easily afford them, and you probably can get old Linux greybeards cheap now that everyone thinks the job description is obsolete.

I'm not surprised at all: that's thanks to the beauty of lock-in effects. Especially all those cloud-specific managed services make it really hard to switch, and even if you are treating the cloud as a pure VM platform, it is considerable work and risk to switch to a self-managed bare metal setup.

Basically, if you don't start bare-metal (incurring having to assemble the necessary know-how to do so) you will have a hard time to get clean from the cloud drug later on, when you think you're ready to afford having the know-how. I think this aspect has been a main driver behind the linked article.

S3 costs pretty much what that infrastructure costs to run yourself.

But certainly, go ahead and run a Riak CS cluster (plus those other things) on top of some Herzberg servers, and try to outcompete S3 on price. I—and many others—will be waiting. :)

And that's ignoring the Glacier storage class (which probably didn't show up on that historical graph), which is a fifth of that.

And Glacier Deep Archive is a further fourth of that.

The latter number can be pricey. But, irrelevant with respect to the failing costs of hard drives.

You may be interested in the 'rclone' tool which does all of these things, and more, with all of the cloud providers:

https://rclone.org/

https://rclone.org/commands/rclone_mount/

$12 million/year for cloud seems like a high enough threshold to go internal but I'm not even sure that's enough money to sway executives away from cloud.

The main issue is that internal IT departments at corporations have slow service from the IT staff compared to AWS/GCP/Azure and its engineers. Corporate IT departments typically don't treat their coworkers in other departments as internal customers. Instead, they treat them as adversaries and a nuisance.

That's why at non-tech corporations, the first group that experimented with the new AWS cloud service were the development teams. They got fed up with IT service backlogs waiting for new dev/sandbox/text servers. They got tired of waiting for IT to requisition a server and then install the os, db, etc. If the programming manager then asked IT to install an Oracle update, the IT department might say "well, Susan our db admin is on vacation till next Monday so we'll get to it then." Those kinds of inter-department interactions were frustrating to the internal customers of the IT department. With AWS, the programmers got their sandbox servers in spun up minutes instead of weeks or months. Once the dev teams were sold, the more risk-averse and mission-critical production workloads eventually moved to AWS too.

The author of this article doesn't seem to understand the dysfunctional relationship between internal IT and their coworkers they serve. It's more about responsiveness and iteration speed of the AWS/GCP/Azure employees' vs the IT employees than expensive AWS EC2 compute power vs cheaper internal racks of cpu.

E.g. The Guardian pays AWS more for cpu+diskspace+network than what they can build on their own. That's the raw hardware costs. But it doesn't matter because Guardian's internal IT staff with an internal cloud stack just can't match AWS: https://www.computerworld.com/article/3427004/the-guardian-g...

To me, it totally makes sense for Dropbox to migrate off of AWS to save money. They're a tech company and have the engineering culture to do it. A lot of non-tech companies (like The Guardian) don't.

That's how it starts but not how it ends.

Once you move everything to the cloud, the same IT staff get put in charge of it. Because they have a requirement from somewhere up high to make sure everything has antivirus on it, and there is a Windows domain for the desktops so the servers need to be in it too, and they want to have a test environment with specific versions of specific software matching the ones in production which means you have to use the corporate VM images instead of the AWS ones and have IT go through their VM provisioning checklist for each new instance, etc.

And some of those reasons are bad and some of them aren't, but either way they exist, so if you want to provision something from AWS now it has to go through IT who takes just as long as they ever did to do what amounts to the same thing as before only now it's more expensive.

Even pre-virtualization, I could instantiate servers very rapidly with network based imaging. With pre-staged templates of various os/program combinations. Applied them to both lab and production deployment. So that Dev, QA, and Prod could all work from the same base images. With versioning of the images based on patch level.

Then I would subsequently get blamed for every program error by the dev team. With no other justification than "it worked in the lab. Fix your server!" Running a packet analyzer, I would often find things like queries trying to go to a lab db server rather than prod. Or a dependency on an out of date/unpatched/vulnerable version of some supporting software that the development team would then attempt to pressure into production rather than update their code.

For every obstinate sysadmin you've encountered, there is an ops pro who has received a ticket with nothing but a list of compiler errors combined with 'pls fix asap'.

One deep problem with the "internal customer" model is that you don't get a larger budget when you serve your "internal customers" better.

But people are REALLY REALLY expensive (not only the ongoing costs, but hiring, firing and overhead), and Cloud has another huge advantage: accounting tricks (which help a lot to said midsize firms).

Literally one of the first things they teach you at the Azure training is the difference between CapEx and OpEx: https://docs.microsoft.com/en-us/learn/paths/azure-fundament...

Ya just gotta know your business needs.

The author's advice is good for basically one scenario: a personal hobby project run by somebody who already knows they would prefer to run their own servers instead of use a cloud host.

The ones that know how hire good people. If a company can’t hold on to talent then they’ll probably fail eventually. Cloud won’t fix bad management.

I have to strongly disagree with this sentiment. Thoughts which are against mainstream deserve to be visible and thus open for debate.

My company for example decided to go with our own production hardware. I'm not saying it's easy. But it's doable. And very educational. =)

One has to remember that running your systems on 3rd party cloud provider will not strip you from responsibility and/or software maintenance cycle which usually presents most of the "human workload". Their nines are not yours.

According to my experience with two reliable machines (redundant PSU's etc) one can create a very reliable service platform. Using correct kind of management tools of course is essential. RAID disk system and a reliable backup arrangement are essential too.

We used Ganeti (https://en.wikipedia.org/wiki/Ganeti) in our 1st generation setup. The next generation will use our own manager (Deux).

Also, good luck finding a Dev Ops engineer for $57,600 a year.

There’s nobody you pay, in your own company, to sit around watching S3 or Lambda or SQS for degradation. That ops service is part of AWS’s job that you pay for through each service’s fee structure.

So the salary we're really talking about actually is less than 40k a year.

Good luck ^^'

From my experience, doing sysadmin for a single server only takes time initially. You might spend a couple weeks full time to get everything going, but 6 months in it's probably only a couple hours a week.

Add downvoting and the article queue would quickly become the same groupthink-swamp that the comments section has become.

One of the nice things about HN is you can still find articles that are against the grain and have topics that challenge the comments section’s “one correct opinion”.

I agree, they are very expensive for what you get at any even modest scale. (Except for s3 which is absolutely amazing). However, if you want to run your own stuff, you're buying into knowing the details of how it works. eg you better understand what vacuum is for pg, and how it interacts with availability, and be willing to create backups, etc. AWS does do real work; it's just expensive.

All I know is we never had a $30,000 “oopsie” in the data center because a dev left a bunch of stuff running they should have shut down.

We aren't going to suddenly have 20x employees one day, a lot of our infrastructure is highly predicable and stable in load and demand.

Some things need instant scaling, sure. But not everything that's in the cloud does.

I know you hate their opinion, but also attacking the character is kinda unnecessary. Can we just not?

https://github.com/jackdoe/txt.black/#start-of-content

https://txt.black/~jack/

In other words this is the OP "zulgan"

user: zulgan created: July 11, 2018 karma: 530 about: https://scrambled-eggs.xyz https://baxx.dev https://github.com/jackdoe

With some cloudflare-ing you could quite easily get away with a single origin server in Germany, too. The latency from a rails app is probably higher anyway.

My anecdata with hetzner - 1.5 decades of lightly used servers for hobby project. One power supply failure resolved in a day. They have some quirks, but.

If you truly have to scale out, sure, cloud is great (sort of). But you can engineer your solution to take the benefit of both, you win.

I think there are valid reasons for each viewpoint.

His "homepage" is a raw directory dump with a high-contrast black background (the preferred color scheme of 10x engineers).

Isn't that enough authority for you?

BTW if anyone knows of more nuanced explorations of Cloud v. DIY tradeoffs -- we all know there's a tipping point in favor of the latter, somewhere -- please do share.

100% agree. This article goes so far as to be farcical, but that doesn't mean there isn't something worthwhile to be had in the conversation... as long as that conversation was framed appropriately. Especially around cloud value-adds like managed databases and Lambda and security technologies like CloudWatch/Trail.

I'm the sole (part-time) dev-ops resource for a very small company and we (by which I mean I) DIY just about everything running on a small VPS on DigitalOcean. I run my own database and manage my own dokku and user access etc. I use S3 and have dipped my toes into Lambda but I always wonder if I'm missing out on something not using cloud technologies more heavily.

I would absolutely love an analysis of what cloud technology companies SHOULD be using and at which point they become or are no longer worthwhile in favor of an alternative.

Honestly? Lots.