This is a myth directly from cloud providers marketing campaigns: that using the cloud simplifies infrastructure to the point that you don't need as much and/or as qualified staff.

In reality it is quite the trap: without experienced system administrators AWS and the like are free to dig their hands deeper and deeper into the pockets of their unsuspecting customers.

I agree the cloud providers like this idea, but I'm not sure I agree it's a myth. Businesses need to choose what competencies to focus on. For nearly every other area, businesses rely on suppliers. Auto manufacturers use machines built by other businesses and parts built by other businesses. They also obviously do some amount of in-house part and tool design. It's a balance and I think that it's a lot more likely that most businesses will want a good partner to provide infrastructure (ultimately at margins less than AWS) instead of building it themselves.

Yeah but they generally aren't renting those machines. Renting critical pieces of business infrastructure that aren't easily replaceable is opening yourself up to rent seeking on the part of your supplier. Remember the Oracle business model--lock people in and then keep raising the price just below what it would cost to rebuild from scratch.

In other examples, many airlines do lease airplanes, and even among the airlines that do own the planes, they still usually lease the engines.

As the GP said, there's a balance to be struck. Delta doesn't have the expertise or the economies of scale necessary to build their own engines from scratch, but they do have the ability to lease those engines and then have their own teams maintain them. Similarly, most companies don't have the know-how or the EoS to run their own data center, so instead they buy cloud services and then have a smaller team to maintain it. It's a win-win. Some companies like BofA might be big enough where they can run their own cloud and it be cheaper than using a cloud provider, but that's definitely not a common thing.

Rental only makes sense if there is a marketplace for used machines. Eg, planes and plane engines are components that aren't really tied to one place or company. (The "that aren't easily replaceable" from the parent post.)

But industrial robots for car manufacturing seems like rather specific hardware. If, say, the Ford plant in Claycomo, Missouri no longer needs a machine for making the F-150, who is going to rent it? How is rental a better cost-savings for Ford than buying the machine and later selling it?

I tried looking this up, but failed. The closest I found was that "over half of industrial robot purchases in North America have been made by automakers" at https://www.robotics.org/blog-article.cfm/The-History-of-Rob... .

Also, '"rent" the maintenance contracts' doesn't make sense to me. I sell maintenance contracts to my customers, I don't rent them, and I don't understand what "rent" could even mean in that context.

The parent comment mentions that Delta rents their airplane engines. I imagine that they have the capability to do general maintenance on the engines, but that the engines need to be periodically sent back to the supplier (probably GE or Rolls-Royce) for a teardown, safety inspection, and rebuild. Without regular inspections an engine can't be flown, and is essentially worthless. Not much point in actually owning the engine if you can't use it.

With that in mind, I can see support contracts as an operational expenditure.

But if "over half of industrial robot purchases in North America have been made by automakers" then that's a lot of capital expenditure. I never got the feel that the companies which make modern industrial robots primarily rented them out (akin to IBM's renting out of tabulation machines 100 years ago). I really thought they mostly sold the robots.

An assembly line isn't really a "product," per se. It's more of amalgamation of several products and technologies, many legacy, that forms a cohesive but loosely-coupled whole. You may have several vendors' products / systems in a line. So what you end up with are ongoing contracts that provide for installation, programming, troubleshooting, maintenance, etc.

There's actually a whole specialized industry that does nothing but design assembly lines. It's pretty neat.

There's Boeing which also used to have an airline. It was made illegal IIRC.

And there are things in between these two extremes.

Some airlines have a heavy maintenance organization. Some have outsourced it. Different companies have different market segments and safety records etc.

The point is, there's no one true way for all.

Of course it's a question of scale, but a lot of arguments I've seen usually come down to CapEx vs OpEx accounting which is silly because a dollar is a dollar.

The cloud is powerful, but for long running businesses who can afford to invest in the competancy it's stupidly expensive. Hybrid cloud installations are ever more the norm, private clouds increasingly capable (PaaS, IaaS, SaaS), and toolsets like Kubernetes reshape how a lot of those legacy apps get looked at.

So no... not necessarily... CapEx is not necessarily paid out all at once. So the accounting ought to be the same whether its a CapEx purchase under a Subscription Agreement/Loan, or if its an "OpEx" payment for 1 month of services.

I mean. You save the headcount in one area and you consume it in another. And you pay overhead in outsourcing costs.

It’s likely worth it. But you need to determine that on a case by case basis.

My (large) company is too incompetent to do hosting properly so cloud was a blessing for us.

At a certain scale, you need all these things with internal clouds too. Resource provisioning, basic security, and managing "insider risk" don't just go away when you own the infrastructure.

This is like saying that "You also need to have a cashier!" when you're comparing a shopping mall to a corner store.

I think it's a loooot easier to design and build a cost-efficient multi-tiered high availability application than it is to make reasonable cost projections in whatever micro-currency every seconary and tertiary service uses.

And I'm ever surprised by how time-intensive it is.

Tooling has only improved.

We were able to cram a large set of applications into a group of instances using CapRover.

Instead of 2-3 instances, and a whole bunch of RDS, we ended just recreating the databases there along with minio for storage (we write a small amount of data to S3 storage).

Colocation is probably ideal for anyone using custom hardware: like GPUs or FPGAs. If you're using "normal" CPUs, just buy dedicated instances instead.

Anyone who has been in the industry since pre-cloud explosion knows that most people just rented bare metal and at the more complicated end they might purchase their own hardware and colocate.

And for those that own their servers, many lease them and HP or Dell or whoever maintains them on site if anything goes wrong.

I'm surprised that more companies aren't playing moneyball with nice, but cheap, locations.

Believe it or not most U.S. programmers don't live in the Bay Area.

You can rent vs buy at any level : rent only machines, rent a 1/2 a rack, rent a full rack, rent a cage, rent 1/2 a dc, and so on.

Same with connectivity: you can plug your machines into a lan managed by the hosting provider, or run your own routers and peer with their in-house ISP, or you can buy peering directly in the building, or you can rent lightpath from a telco with pop in the building, or you can go rent a backhoe and start trenching across the parking lot...

Cloud is awesome for companies where the capital investment needed to deliver the SLA is too expensive.

If you’re big, and you run the numbers, lots of services are better in a datacenter that you manage. Many of the early advantages of cloud tools aren’t a differentiator today. If anything, the granular billing is a nightmare to manage in many large enterprises. There’s a reason AWS heavily targets .gov customers — the Feds probably waste billions on idle services.

I’ve found that the shitty to support or known workload services are best somebody else’s problem. Email is an example of both. Most other things are trading IT complexity with accounting complexity.

Maybe cloud usage can be an indicator whether a company has plans to maintain independence into the future, despite the loyalty-engendering words of leadership.

Situation: some manager is tasked with updating some antiquated COBOL monstrosity. They're given twelve peanuts, some shoe string, bubble gum and a little bit of duct tape. The experienced staff have all been aged out, quit out of frustration or laid off and replaced with fresh college grads for half the price which was a great deal and significantly reduced costs for several consecutive quarters, earning a well deserved healthy bonus. Unfortunately reliability of the system has been suffering and the young engineering team unanimously agrees the old system should be replaced.

Option 1: do it in house with the young team of rockstars the manager has been bragging to management about. Unfortunately, the manager secretly has reservations and has doubts about the team's experience, let alone skills. It's too risky. Not too risky to the company who can easily absorb the loss and move on, but too risky for the middle manager who will look bad and possibly get fired.

Option 2: outsource the entire thing to IBM or Oracle. Tempting, but not in the budget. Also, significant parts of the old system are on IBM anyway and the cost of hiring IBM people to fly out has been increasingly expensive after the latest round off layoffs saw Scott put out to pasture. No, it's too risky. Not risky to the company who can have uptime guarantees in the contract, but risky to the manager who will look bad if he comes in over budget.

Option 3: The Cloud. It's the 'in' thing. FAANG are all all in on it, and if it works for them it could work for us. Much of the new team used AWS for their web dev class in college, so the skills are fresh. There's very little risk to it. The staffing requirements will be dramatically reduced so costs will drop. The manager will be able to deliver an awesome demo next quarter and the S3 instance they'll host it on will only cost one or two of their peanuts. The only risk is if the manager doesn't get promoted before the problems start, which shouldn't be a problem because AWS has such great onboarding programs. It is slightly more risky for the company though, who may find themselves no longer in control of critical infrastructure.

Risk for the company isn't the same as risk for a mid level decision maker. Small fuckups carry the same penalty as huge fuckups. (you get fired) From my fake story, doing it in house has a relatively high chance of being a minor fuckup, and AWS has a relatively low chance of being a major fuckup. The manager is going to choose the low probability risk, even though the low impact risk is the company's best interest.

And let's be honest... a remarkably large fraction of small IT shops are really bad at certain basic competencies, notably backup reliability and timely application of updates. AWS is pretty good at sysadmin and accounting 101, even if it stumbles at 201.

Insightful. Can you clarify what you mean by "low probability risk" and "low impact risk" ?

impact risk - Odds of a project failure causing large impact to company.

There are two things worth considering. The BOA cloud is slow. That is to say, it is hard to procure machines to get your projects going. There is a lot of control on who will be paying for them. There is also A LOT of staff to manage it. So those 2 Billion could easily be tweaked downward when you consider the staff and lost of agility.

On the flip side, AWS is hella expensive. To do the things that they do it would cost A LOT of money. Maybe even more than they think. I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills. So this number can also be tweaked up when you consider dozens of teams each having their way (at BOA scale) on the AWS console.

But I would expect that a large established company would do better to create it's own reasonably priced reasonably managed infrastructure.

Sort of rent vs build/own.

And something that seemingly everyone forgets: it's not black and white... you can have some things in the cloud while you have some thing on-prem, leveraging scalable infrastructure and redundancy while also keeping your core assets fully under your control. It's pretty easy to make an internal S3 replacement, it's hard to make a better S3.

I think some companies, like BOA, look at this as the safer alternative.

We all assume that large Cloud providers are bulletproof. That's just because we haven't seen them taken down yet. Someday we will. Just wait until there's another major world conflict. These global companies with outsourced IT will be in ruins because politics and digital DMZ's will keep them from acting cohesively.

For an organization like a bank that's worth over $400b that layer of staffing, equipment, oversight, and all the inefficiencies are a $400b insurance policy that you will be able to continue operating regardless of geographic or political events outside of your control.

I've been at multiple Fortune 50 companies at multiple levels. Many times these types of infrastructure building are just forms of empire building and an attempt to embed ones' services in the organization at a very deep level. You convert one set of costs (AWS/GCP/Azure bills) for another set of costs (Infrastructure + Payroll). You also convert one set of headaches (CSP overcharging/auditing) for another (prima donna internal infra engineers.)

Don't get me wrong, most corporate IT is deeply incompetent at their core job. It's just that you can't fix the deep problems by outsourcing the computer part.

There's this great thing about BOA-sized support contracts where you get to call up the senior product managers and demand that they build the features that you want. And if you're not happy about something, engineers will drop what they're working on to appease you.

I wonder how many customers were interested in setting up a Custom Keystore for KMS on CloudHSM?

https://aws.amazon.com/blogs/security/are-kms-custom-key-sto...

Of course, for small or medium customers whose finger snaps are ignored... WYSIWYG. The roadblocks in not-your cloud are still your roadblocks.

But (in a vacuum) when developing a solution to serve diverse needs and stakeholders, the HiPPO effect is definitely an anti-pattern.

With CSPs, at least you have four major players (AWS, GCP, Azure, Oracle Cloud, ?IBM?) competing prices downward and multiple theoretical options. Yes, you have your in-house CSP expert biasing things, but since most of the work is external, at least you dont have a heavy internal weight on a home-built choice.

IMHO, the better option would be a CSP-agnostic Kubernetes based layer. But IMHO we're not there yet.

It was about decision maker's personal (career) risk when making decisions. Keeping things in-house means more decisions are taken in-house which increases personal risk to the management for getting things wrong. In context this is much clearer:

> This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions (which can be career-costly, regardless of right decisions made previously).

The post above doesn't discuss cloud Vs. non-cloud security/safety at all.

To your point, we need only point to recent events to show the universal customer pain that can be caused by a bad code push at Azure or AWS.

If a company (or person) really wants complete visibility and control into their relationship with law enforcement, they have to host their own hardware... ideally, their own data center.

I think the opposite will happen. Cloud providers are becoming commoditized. Kubernetes, blah blah. Developers hate lock-in, and on a long time scale, developers drive tech decisions. Nowadays new software is cloud agnostic, and it will continue to be that way. This is commoditization, which generally drives down prices.

That said, buying commodities from a market with four participants is more like buying from a cartel than a marketplace. I definitely worry about the consolidation of the world’s IT spending going into the coffers of three or four massively profitable and morally corrupt corporations.

You and enterprise software should get together and have a introductory meeting...

There is absolutely value to delivering a product to market a year or more sooner than otherwise.

Separate from all of that there is a valid question whether the software stack at a big cloud provider gives your engineering department the ability to deliver features faster at scale and redundancy, and with lower ongoing maintenance overhead. I think this question is very much up for debate.

At a small scale there are absolutely cloud frameworks that let you deploy services without an IT department at large scale. But then there’s a realistic argument of whether the high premium for theoretical scale—when in actuality you’re serving dozens not millions of customers—is worth the high cost of capital.

E.g. several startups of mine have persistent on a couple colocated dedicated servers for over a decade at a cost of $300/mo. The equivalent deployment in the cloud is closer to $3,000/mo.

I know this because I got $300k of Azure credits to play around with and ended up not using most of them. The long-run cost was too high to move to Azure. But there were months when I “spent” $6k of credits on a dozen VMs and associated storage and network that had equivalent performance to my rented bare metal.

I’ve encountered quite a few devs whose only focus is on achieving the specified goal without considering the full security scope. I don’t blame them, that’s what they are there for. Stepping in for ensuring the customer security and regulation requirements is what I’m here for.

It's like using package managers like Maven and npm. Sure, they make going to market much faster for small organizations. But at companies that have strict security procedures, downloading unvetted stuff from the internets and putting it into production does not fly. You'll have internal repos that work with your package manager tools, but nothing gets into those repos without at least a cursory review.

The two companies I've used AWS at both operated this way. In fact, each team has multiple product accounts (for dev, testing, production). So I don't think it's unusual at all.

I've always wondered about this. In IT now instead of having 5 system operators one has 10 "cloud consultants" and nobody knows what they are doing. I've been through a couple of "migrations to the cloud" now and I've never seen a reduction in IT staff or the cost of running IT operations.

About the wrong decisioning part, if you chose the wrong "cloud" for your business the cost to change can be enormous.

The primary motivation for management is from what I've seen so far the ability to point to someone else if shit hits the fan, not cost reduction and all that other stuff.

You start wondering if 90% of corporate world are there to do nothing if not erecting roadblocks for people who are actually doing something. But then you start wondering maybe it's a good thing that all of these people get paid and could spend their money so that majority of the country has purchasing power and not live in piss poor poverty.

For example if I can project steady growth or decline of my needs, it seems that I benefit less from the infrastructure-on-demand like AWS/Azure/GC

Also, depending on the size of the organization it seems that there at least several options on data centers:

  - have my own data center
  - collocate my hardware in other's data center
  - rent hardware
  - use infrastructure-on-demand + software-as-a-service

Many organizations deploy Pivotal's https://pivotal.io/industries/financial-services

or Redhat/OpenStack

So they are not actually 'building' the infrastructure management layer from scratch.

Also for investment banks (due to exchange connectivity) , telecoms (due to peering points, physical tower locations) always going to have their 'Front-office' applications in data centers that offer lowest latency for those apps.

Midoffice applications (or BOSS as they are labeled in telecom) can live in data centers reasonably distant (in terms of ping time) from the front office. So the infrastructure for those has a different cost profile.

Most of this applications touch trades or trade events that carry millions of dollars worth of contract value (not Facebook's likes or favorites... )

There is a lot of logging, reconciliation, post executing enrichment that's probably going on.

Finally, from my 2nd hand understanding,

Investment banks can have from 3K to 10K applications in production. With, perhaps 20% of them having more than million lines of code.

Some of those apps are built in their own proprietray languages developed specifically to manage contracts and lifecycle of contracts (Banks had 'smart contracts' DSL and interpreters since 90s).

https://news.efinancialcareers.com/uk-en/282097/slang-goldma...

A company like BoA will eventually choke until this in the future.

Also, this stuff sounds great until you need improved peering, or software-defined networks or massive bandwidth improvements, etc.

... and I get that the HN community is clever but banks and other institutions like this REALLY struggle to find this talent.

$2B is a lot except when you're losing $10B a year in revenue because you can't compete in the market.

In a way this would be like Amazon arguing it should build its own bank. It's not what they're good at... Focus on your strengths.