Currently I work in a company that's very cost-conscious about hosting. They are generous in salaries but are mercilessly practical about IT costs and the abilities of computers -- especially the latter part is something that most of the programmers today have almost completely forgot.

I have 3 spare laptops lying around; weakest of them is with a Celeron J4155 and I have put a web app with no caching on it (it does have a persistent DB) and hammered it with my workstation until it finally started giving up at ~2500 req/s (Elixir/Phoenix stack). Again, that's a Celeron J4155 with a SATA III SSD in an M.2 factor (so disk speed caps at 550MB/s at best; usually 400-460) and 12GB RAM. Most programmers wouldn't touch such a machine.

I imagine I can buy 2 more of these laptops and make a completely replicated 3-cluster of the entire stack of our company and the slowest requests (on admin UI where we have a lot of SQL JOINs) would likely never go above 200ms. That totals at about 600 EUR (yep, I bought the laptop second-hand for 200 EUR). Then 500 more EUR for a good UPS to plug the laptops and my routers to. Boom 1100 EUR and several weekends later I can likely charge my own employer for hosting at 100 EUR a month for their entire infrastructure and I would likely still be ripping them off even with that.

The only real cost is human time and energy invested in making it work. But for most companies that's not a 24/7 fight so that cost is fairly low. You can do it twice a year and you're likely never going to have problems.

So yep, I am completely with you here (if my rant didn't make it obvious). Infrastructure costs are already being heavily optimized by companies out there.

> The only real cost is human time and energy invested in making it work. But for most companies that's not a 24/7 fight so that cost is fairly low. You can do it twice a year and you're likely never going to have problems.

If this were true, what made the cloud providers popular in the first place?

Outsourcing culture. Nobody wants to nurture talent -- that would also mean to invest in relations with your employees and not alienating them. Shareholders prefer to sacrifice a little more of their profit so they deal with less potential problems. And the pull of the idea that every human in the org must be an inter-changeable cog is too strong (even if that idea continues to be absurd, and always was).

Hosting apps in the cloud was a fair exchange 10 years ago because operational tooling in general was more immature. Nowadays it's much easier to self-host many pieces of software though.

This isn’t it, you might not know or if you haven’t been in an environment like it.

Without a cloud you’re always running up against limits, out of power, out out cooling, out of rack space, out of hardware. You get new resources by adding to wish lists and seeing if the end of quarter budget will agree with your request which might be filled in a few months, maybe next year, often never.

You hoard hardware that ends up doing nothing most of the time so you have it when you do need it. Management spends a lot of time and energy managing the datacenter budget.

With cloud you get what you want without asking too much and management periodically spearheads savings efforts to show off, but ultimately usually spends a lot more than they would have otherwise with less friction.

A big part of cloud adoption, according to my theory, is getting executives out of the way of computing resource needs and freeing up their time to fill with something else like bothering employees for more status updates (which are easier and require less skill).

> Without a cloud you’re always running up against limits, out of power, out out cooling, out of rack space, out of hardware.

I grew up in this era and keep hearing this repeated but it simply wasn’t true. Enterprises would plan ahead and buy enough hardware for years and it would work fine until you bought more. The myth that you need to scale your infrastructure 10x in a day doesn’t apply to 99% of enterprises, and even if it did it’s probably a result of bad planning on the part of leadership. As a result of the current paradigm businesses end up renting servers at a substantial markup for fairly obsolete hardware.

In general it's really the opposite. In 2004 you needed a rack full of $3000 servers to run your medium business. Now it's two physical machines using 5% of the power to virtualize everything that used to run on two dozen.

Over a given period of time, computers get faster/cheaper by more than most businesses expand. When you need to expand, buying a newer, faster machine may cause you to save money because the faster machine uses less power than the existing one.

You are talking big businesses here, I've seen people just ask the CEO if they can buy 3 brand new servers and him agreeing, verbally, and the servers arrived next week, and two weeks later were completely setup and were useful.

This lasted for 11 years and only stopped because two of the 5 senior engineers retired and because the company was bought a few weeks earlier.

So again, don't look at this through Silicon Valley lens. Most of the companies in the world have a very different mold compared to SV.

Then there are service contracts to make surr the systems keeps running with little downtime. Which may be impossible to get for 11 year old equipment. Then there are pesky details like needing a disaster recovery site.

Professional hardware is expensive and server h/w is a small part of it.

> Without a cloud you’re always running up against limits, out of power, out out cooling, out of rack space, out of hardware.

I bet most customers of cloud services are not in a high-growth phase, so this is scenario most organizations aspire to ("What if we suddenly got popular?" is a fantasy that's hard to disabuse someone of internally, if you want to be known as a team player

> Management spends a lot of time and energy managing the datacenter budget. With cloud you get what you want without asking too much...

I fully agree, this is the core reason why most companies gravitate towards cloud: management abdicates control of costs to engineers, resulting in less friction - but its OpEx, not CapEx, so the bean counters are chilled about it. If the same low-friction approach were applied to DC equipment, you'd get similar results, but cheaper.

I used to be a big proponent of self-hosting. The raw hardware cost makes it look like a great deal. However in my experience hardware is less than 5% of salary cost and having enough admins on hand to make your infra reliable is usually going to end up costing you more in the end.

Don't some of those salary costs transfer over to the cloud. Someone still has to manage it

You need a lot less people per computer when you have millions of computers like big tech has than when you have 10.

The cost of these services is not because big tech has to use that much to run them, but because big tech would make less money if they lowered prices. AWS generates tons of profits, why lower that for no reason?

> You need a lot less people per computer when you have millions of computers like big tech has than when you have 10.

That's not the people we're talking about. Racking a server and setting it up to do virtualization takes maybe a few hours for one person, if that, over a period of years. Maintenance on the host itself is the same.

The real labor cost is in setting up and maintaining applications for your specific needs. None of that goes away by using someone else's hardware.

> The real labor cost is in setting up and maintaining applications for your specific needs. None of that goes away by using someone else's hardware.

But AWS does a lot of that for you by offering cloud services and not just hardware. That is why people pay so much more for AWS than other just hosting solutions.

That doesn't explain why their pricing for generic VMs or bandwidth is so high, or why anybody should want to pay them for that.

It also doesn't really work. Some textile company is going to have some line of business software to run their textile mill. Amazon doesn't provide that. You still have to do labor to configure it. These are the hard things, because they're custom and don't have a huge installed base of people who already encountered and solved all the problems you're going to have. But for the same reason, they're the things AWS doesn't provide.

What they provide is common things like DNS. But DNS is easy to set up and maintain, because it's common, and so already has smooth edges. That's not where the labor was going.

> Outsourcing culture. Nobody wants to nurture talent.

Also, outsourcing moves the blame to someone else if things go wrong. (and things in infra go wrong nearly constantly).

The problem with this kind of thinking is ofcourse, that there is no risk taking and innovation in suchs an organisation..

AWS lets me provision servers all over the world at any point, which is required by customers both for compliance and for latency SLAs. Even billion dollar companies are going to find that a huge lift to build out themselves, and they have opportunity cost while they try to do it.

What about scaling down? Once you bought hardware for self hosting you are stuck with it.

What about it? We're talking small businesses here, this will not be a warehouse full of racks worth of machines to try and sell. We're likely talking 3 to 10 PC-sized machines in a closet or storage room. You can sell those on 2nd hand market pretty quickly and even if you sell them at 30-40% loss you are extremely likely to have already paid off the other 60-70% during the machines' service time.

old laptop is fine then :)

The costs the OP described are pretty minimal.

You never know. While working in an ITAM role, I've had a manager point out a line item asking why we need 30 DisplayPort cables. This was in an org with multiple sites, 2000+ employees, using dual monitor setups both in house and remote. Whether physically, or in the cloud, companies whose primary business isn't tech-related see the IT side as an annoying cost center rather than a cost of doing business in the 21st century.

This selfish outsourcing culture is killing all the Gilfoyles out there!

I'm curious as a hosting and cloud noob, do you have an estimate how much that would cost monthly or yearly if hosted on a cloud?

Not really, because AWS costs aren't transparent. You can't drill down properly and even where you can, the prices are disappointingly high.

It gets really tempting to setup a backup/failover node on one of my spare laptops lately...

Interesting. I am not intimately familiar with the subject matter, so I will be curious then to see how the next 10 years play out.

Beancounters love OPEX and avoid CAPEX

beancounters are not in charge of the direction of the company. In a lot of cases, it makes a ton of sense to go for capex vs opex.

My old employer had a strategy which basically boiled down to owning everything inside this company expect for the coffee machines and cleaning crew. His reason being? This made it possible to run on very thight margins when the economy takes a downturn without having to scramble for money because of leased/loaned equipment etc.

In the 2008 crisis, this is how he stayed afloat with his company, and even made a pretty profit during a time of crisis too.

Surely the beancounters can also see the effects of capex on the valuations of the cloud providers.

If they're publicly traded, no one's going to give Bob's Consolidated Widgets a 65x PE or 2x+ PEG ratio, even if they have the best IT strategy in their market.

Curious, can you explain why? In my experience (just one company where I had some interaction with accounting), there was a preference toward CAPEX. The costs of building a product, ie software development, was prefered over OPEX, in this case, analytics. Since I was doing both development and analytics, they preferred me to account part of my time as CAPEX. Not sure if this is normal but for context it was a non/pre-public company.

OPEX is just an expense in the current fiscal year. CAPEX depreciates over N years. This alone means that a one-time payment is much simpler on the books.

I don't think your salary counts as CAPEX, it's a normal monthly expense for the company.

Keep in mind that most big clients of cloud providers are startups running on VC money. It's not their money and they don't care about wasting it, and VCs don't particularly care either.

This has now created an entire ecosystem of developers and "devops engineers" (what we used to call sysadmins) who know little beyond the cloud and have to keep using it for career-related reasons. This in turn pushes companies to use the cloud as finding talent for old-school on-premises infrastructure is difficult.

Infinite scale up (direct fan-out to mass market), high margins (able to overpay for commodities), and outsourcing production work to stable businesses are all facets of the Hollywood business model that Surveillance Valley adopted. Once the concept was socially proven, then nobody gets fired for buying IBM.

That human time happens to be quite expensive.

Not in small companies.

It's most of the cost at small companies. A lot of people underestimate the total cost of having employees perform tasks at unpredictable scale with unpredictable problems versus just writing a check each month.

It's true that cloud providers aren't reliably cheaper as some assumed at one point. But there's something to be said for cutting out the fully burdened costs of some number of employees (especially given everyone is saying tech talent is expensive and scarce at the moment) and just letting AWS deal with it even if you could theoretically do it cheaper yourself.

Not only that, but if you have a business that actually makes, sells, and ships something, your AWS bill is probably a tiny fraction of your expenses. Switching to physical servers would be a little like getting solar installed on your house - great in theory, but if the most expense it will displace is $150/month, is it the right problem to focus on?

True. Although I’d argue most companies are small enough that AWS itself represents more overhead than is optimal. They’d be better of on Heroku, or even just a single VM.

Certainly you can overcomplicate things with any of the big public cloud providers. As you say, many smaller companies would be better off with a SaaS, maybe a PaaS, and some VPSs, whether from one of the big guys or someone else.

I had to laugh at someone's comment around re:Invent last week that just because Amazon has to offer 17 types of databases for its customers (or whatever the exact quote was) doesn't mean you have to collect them all as if they were Pokemon.

What if the cheap guy quits?

I'm aware of the risks. That's why I said a culture of long-term relationships must be fostered if you want to pull such operations off.

Where I work right now people don't just quit two days later. They like the company and if they feel they want to go someplace else they're not being difficult about it. They cooperate in passing down their knowledge to colleagues before they go.

You are vastly overestimating the IT capabilities of most companies. The human time and energy cost that you handwaved away is a real killer because that is expert knowledge capability that is highly valued by the market. Maintaining an IT staff and infrastructure to support a large organization that has to perform any kind of reasonable SLA for services eventually puts you in direct competition with Major Cloud Providers at scale. A scenario where you can outcompete them with their large customer bases becomes very difficult to imagine under current technological and market conditions.

It's not a matter of outsourcing culture. Cloud providers really do provide computing in a cost efficient matter since they practically provide IT infrastructure at wholesale, and it produces an anti-trust risk because, as you pointed out, the actual hardware that can make it go is not nearly as expensive.

I am not handwaving anything away. We're a company that has less but very strong programmers (we don't even have DevOps / sysadmins, our CTO is doing it and I'll likely be taking part of his responsibilities at one point).

And we don't have the requirements of most big Silicon Valley companies that you reference. It boggles my mind why so many commenters IMMEDIATELY assumed we're a hyper-growth startup or whatever. We're an expanding data provider and our growth is super predictable, and our set of requirements changes like once a year.

So really, not sure why you and others keep repeating things I already agree with. The cloud wins from one point / scale and on, absolutely; otherwise it wouldn't ever take off. I am saying that people give up way too quickly and run to the cloud long before they need it -- that's all.

If running your tech stack on local hardware is right for you guys, than more power to you. That is also what works for me for my personal and hobby computing requirements, where I maintain a home machine. But there is kind of an implication that most people who start developing on cloud services are just clueless programmers who never learned the basics of system provisioning.

Even with the specs you mentioned, you can rent a lightsail server that is comparable on AWS for like 80 usd a month, so I would push back against the idea that using cloud services in pretty much any scenario is a purely technical decision.

Part of the context of discussing things this way is the original article: do the prevalence of service companies signal the end of big tech? I would say no, because the only way to not be reliant cloud service providers is to do what you describe and compete directly with their infrastructure services by forming your own technical infrastructure. But the logical conclusion to doing things this way is that eventually whoever can control more physical infrastructure has an advantage. What if Amazon is able to replicate your service offerings? They would be able to immediately able to offer their version using their cloud services at a fraction of the price because they can amoratize the cost of computing over their cloud business. Then, when they have eliminated you as competition, they can recoup everything by increasing the price. This is a HUGE anti-trust concern, and one that all tech companies should be at least a little bit concerned about in the long term. It doesn't go away because you can have an (in your case more expensive than lightsail) up front investment to own your companies computing resources.

If they can do it at the fraction of the cost then I'd use it. Trouble is, AWS pricing is hard to parse, especially post factum. In my company's case, the leaders are not happy with the lack of billing transparency.

I think that's not only a good reason not to use AWS, but also the fundamental problem with the position that cloud service providers pose. It would be much healthier for companies and organizations to directly own their own computing resources, rather than effectively leasing them from one of three cloud service providers, but it is absolutely more of an investment for companies to make.

Sure, agreed. I am on the opinion that Amazon does not have to be that monopoly-oriented but alas, humans being humans right?

Give me an AWS service with predictable billing and ability to put hard limits so you don't have your credit card charged for surprising amounts if you get bursty unexpected loads, and I'll be all over them without thinking of on-prem hosting.

Honestly, I don't even think the billing is that untransparent - all their services have pretty straightforward pricing guidelines, and certain services are load limited (going to point out light sail again). It's pretty fair - you pay for what you use, and that works a lot better for most cases if you know what you are doing, you just take on a bit more risk that you have to guard against.

It would be a lot preferable for there to just be multiple computing resource providers, rather than just three, but the reality is that setting yourself up as a cloud provider is quite inaccessible. Democratizing the industry of cloud services will probably require regulation.

But you yourself admit that you have an unusual setup with a few very strong programmers. Few are like this even in Silicon Valley. As for other industries, they're not even close. So your company's setup won't transfer well to the rest of the world.

I would not be so sure of your last. The SV mold is the outlier but most USA programmers can't see it because they don't work for non-US companies. Everywhere I was all around the EU in my 20 years of career so far, the SV way of doing things has not been the norm.

So it pays off to be wary of filter bubbles. The SV way of doing things assumes a ton of VC money to burn, something that, to this day, is still not a wide trend in the EU (even if it happens here and there).

Plus that company's setup is not unusual in any way per se; the only "unusual" thing is "we don't run to the cloud at the first IT problem". A ton of other companies are like this, I'd bet even in the US.

What happens when your local internet connection goes out and you can no longer serve traffic for your business? What if there is a power outage? Who will spend the time replacing disks as they fail? Where is the data backed up to, so that a failed disk doesn't lose business data?

I like where you're going with this but you could take it a lot further with little expense. Amazon and MicroCenter have a huge stock of refurbished HP mini PCs with i5s, 16gb of ram and an M2 slot for about $350. Throw in a small m2 ssd in each (~$50) and keep your UPS, that's about $1700 you have a decent cluster for an efficient stack like phoenix.

How much money would your company save by hosting on some old laptops instead of in the cloud?

5 digits of USD per month. Whereas normal on-prem hosting can be started with almost no dollar cost, tomorrow.

Right now they/we can't do it though, the talent is there but that talent is super busy reworking software that's no longer well-adapted to their new requirements. We'll see if the topic becomes more relevant in the future.

Not to forget - there are more choices than big-cloud and host-everything-yourself.

One can rent VMs from Hetzner at 1/3 the price of AWS (and 1/10 the data transfer fees). No managed services, like Dynamo or S3, but will full convenience of on-demand scale-up

Or, for mid-sized corporations - buying a rack of computers, wiring them together, and installing OpenStack to make an internal cloud - that isn't some black magic wizardry either

Oh yeah, completely agreed. As mentioned in another comment we might just go Hetzner and buy a singular server in another provider for backup and failover. We'd likely be just fine with 6-7 of the smaller Hetzner servers, too.

It seems crazy that you are paying that much for hosting something that can be self-hosted on an old laptop.

that works as long as there are enough of you for every team/app/service/whatever to build and operate all that around the clock and without bus risk.

A lot of these threads end up talking about "scaling" from a pure hardware perspective. Once you start talking about medium-to-large companies, the real scaling headache is people, process, and organizations.

Not sure if I made myself clear but I definitely meant my above as aimed at small businesses (which my current employer is).

Obviously from one scale and on the cloud is very much worth it. IMO the discussion has to be shifted to "before which point you can easily get away without the cloud?".

At some point, an internal cloud starts being interesting and gives a clear org boundary, using cloud-based software and interfaces.

That's true as well. It all depends on how much is the employer willing to invest in their employees in the end; if they foster longer-lasting relationships then many interesting internal projects become feasible or even desirable.

Those invested-in employees will probably still leave, especially if they're very good and you can't match big tech salaries. And, in any case, you need redundancies for when they're on vacation, out sick, etc. Not to say you shouldn't develop in house competencies but you should do so deliberately because there are probably a lot more costs and risks than are obvious.

We're not blind to the tradeoffs. It's a long process to both change culture and nurture the necessary skills in your staff, AND take care of having human redundancy.

My point never was to idolize on-prem; I have agreed in other comments that from one point and on the cloud absolutely wins.

I'm mostly pointing out that there is a lot you can do before bowing your head to Amazon and accept $30_000+ monthly bill for infrastructure that I can fit in my 1 square meter food closet and which would likely cost me (or a few other experienced backenders) 2-3 weekends to setup.

Absolutely. I'm definitely not in the "You're crazy if you don't just use AWS/GCP/Azure" camp. I'm just also not in the "See how cheap it is to just stick a system under my desk" camp. Totally dependent on use case and circumstances.

True. It's not a religion or a credo. We're simply evaluating our options because our cloud bill is way too high for what we do.

Time will tell but IMO it's super important to reserve the right to take a deeper look if you don't like your costs.

We're not as helpless as big tech wants us to believe.

And figuring out those costs can be really hard. People like Corey Quinn have built a whole business out of helping people make sense of their AWS bills.

Yep, but it's all self-interest. At one point you ask yourself: must I really go down this rabbit hole?

>I can fit in my 1 square meter food closet and which would likely cost me (or a few other experienced backenders) 2-3 weekends to setup.

I don't think that setting it up is main problem here. Keeping it up to date and secure is the hard part.

IMO the thing to watch for is regulations around what rules you need to follow to be compliant in how you operate your data center or physical servers. It’s very likely big tech will lobby for rules that appear to be anti big tech but actually will make it harder to operate independent from a big tech provider… think net neutrality (not saying I don’t think we need it just as an example)

>I expect some startups in near future to target this opportunity

https://oxide.computer

But most of the problem isn't with infrastructure. Once you start using AWS's services it is the software that is most problematic. And that is why all these infrastructure / cloud provider wants to do serverless.

Right now there is a lot of money to be made in moving companies to the cloud, I predict that in 10-20 years time there will be a lot of money to be made moving them out of that same cloud.

I hate having to learn all of AWS, GCP and Azure proprietary tech, I’ll take a Linode box any day thank you

I believe this has been stated to be the core thesis behind Oxide Computer Company's business model.

but this is already the case for a LOT of companies. HN seems so focused on cloud and sillicon valley that they forget there are hundreds of datacenters filled with colocated space and connectivity.

Colo is cheap, connectivity is cheaper then it has ever been, and most bussiness don't require massive scaling to do their bussiness.

Also, owning your own infra allows you to innovate and play a game outside of the borders aws, gcp and azure are drawing up for you.

On a related note, I run noscript on my browser and overall it's not a bad experience. The conveniences IMO outweigh the inconveniences.

However, Cloudfront, JSDeliver, etc are a pain. I understand the benefits of caching javascript locally to me, but I want to see companies start hosting their scripts all from their own servers.

If FontAwesome isn't already a tracking company, it's only a matter of time. Cringy-cool name, tentacles throughout the web, all for _fonts_. I'm a senior fullstack dev and I get tradeoffs, but the technical-societal-tradeoffs of the web are bewildering.

I was thinking the exact same thing recently. If you can setup and manage an infrastructure by yourself, you have pretty good advantage in term of cost.

Why? Especially for small and medium businesses, the capital costs involved rarely make sense, and if you're leasing, might as well lease from a cloud provider which saves you time and people. There are many smaller players which are OK depending on uptime requirements - OVHCloud, Digital Ocean, Hetzner, Scaleway, Linode, etc.

The capital costs are very easy to make back. For us, the payback period was <1 year, and our machines provided unbeatable performance with very good transparency into issues, which made it relatively easy to track down issues. Dedicated machines are really, really powerful now, and you can do with a few what it used to take racks. I feel like the perception of “it’s too hard” or “it’s actually more expensive” is subtly encouraged and reinforced by the myriad companies that stand to benefit - all those hosted services, the cloud providers, etc. And they make it easy for individual employees to appear more productive, but in exchange for costing their companies much more in the long run, and introducing a lot of inter-service latency and complexity. Misaligned incentives.

The major exception for us is S3, scaling storage seems like it would suck terribly.

>...is subtly encouraged and reinforced by the myriad companies that stand to benefit..."

Not just companies. Web developers love the complexity of their stack. Some monolith written in native performant language and put on a decent hardware can cover needs of the most reasonable size business. This simple fact makes them very uncomfy when presented. Many of them would not be even aware that such things are possible. They'd rather be spoon fed by the likes of the Amazon telling them how to develop their wares. The fact that the more inefficient their software is the more money said Amazon will make is ignored. They'd invent all kinds of largely BS arguments to avoid going a simpler route. One can just read numerous articles on HN describing "our stack". On a client site they have this React atrocity. Sure this thing works well for orgs the size of FB. But they're not FB and never will be.

this happens lower down the stack too.

Distributed redundancy is a good example. There are very, very few situation in which this couldn't be resolved with a simple BGP daemon on a server and anycast.

I don't disagree with you but I wonder if you are underestimating how many years of useful service can you get out of dual Xeon E-2690v3 systems. Almost no business actually needs the latest and greatest -- the only serious concern when hosting on-prem is energy consumption and heat dissipation. If that can be covered well the rest is much easier to cope with.

I do have some server hardware on premises (well it is in my basement and I have fat internet pipes) but I do not mind going middle road and renting dedicated servers on Hetzner and OVH. They're surprisingly cheap for whet they offer. The same performance / resource wise hardware on Amazon would make one bleed the money.

Agreed, and that's just what we might end up doing btw. A bit more manual sysadmin work which we end up doing on EC2 instances currently anyway. And a lot of money can be saved by EU dedicated server hosting.

Are any companies selling a package to make this happen. Your Own Cloud Starter kit only $1.5M