News for hackers. Stuff that matters.
Re:FedEx to close data centers, retire mainframes by chrisseaton on Tuesday July 05, 2022 @06:56AM (#31986680)
You can say the same thing for hundreds of parts of a company like FedEx.

For example FedEx flies Boeing jets. It's completely reliant on Boeing for parts for those jets. Presumably Boeing can charge whatever it wants.

Your company is always going to be reliant on other suppliers and contracts. Unless you're planning on building your own independent country in some location that has all the natural resources you need.

Non-argument.

Re:FedEx to close data centers, retire mainframes by tpetry on Tuesday July 05, 2022 @07:02AM (#31986719)
> For example FedEx flies Boeing jets. It's completely reliant on Boeing for parts for those jets. Presumably Boeing can charge whatever it wants.

Nobody said that FedEx should build their own server hardware. And that's what you are comparing it too.

FedEx could just buy server appliances from e.g. Dell (buying the Boeing jet) and operating it. Because paying some other air cargo company will eat a lot of their margin, the same with Cloud infrastructure. They are not a startup which could better invest their time in development, they can without any problems hire some people to administer their server fleet. When they switch to a cloud they will likewise hire some engineers only managing e.g. AWS to administer it.

Re:FedEx to close data centers, retire mainframes by int0x2e on Tuesday July 05, 2022 @07:57AM (#31987122)
I'm simplifying here, but it feels to me like you are making the assumption that FedEx is a mostly static business, whose IT needs should be all about minimizing the cost per IO or compute operation. In the real world, business needs are rarely static, and moving fast and innovating is extremely valuable, even for a company as large as FedEx. They are choosing to move resources from managing their IT infra to AWS, but what they're really gaining is not a reduction in labor costs or CAPEX, but rather the ability to move faster. Sure - given some headcount and sufficient CAPEX, a good engineering+SRE team can create and maintain a nice bit of infrastructure, but it is a significantly harder goal to truly deliver the benefit most leading cloud providers can provide an engineering org.
Re:FedEx to close data centers, retire mainframes by kragen on Tuesday July 05, 2022 @08:04AM (#31987168)
Probably the right amount of rented data center capacity for FedEx is not zero, yes. But it's not 100%, either, because what they're giving up is the ability to move faster when their outsourced system administration vendor isn't meeting their needs.
Re:FedEx to close data centers, retire mainframes by CHY872 on Tuesday July 05, 2022 @11:02AM (#31989040)
Most large companies do at least some elements of multicloud. This may be as simple as doing ETL in AWS and then pushing the data over to BigQuery for dashboarding, it might be using all AWS and also Office365, it might be building applications which part run against Dynamo and part run against Cloud Spanner. It's also the case that the applications you run each year have some turnover rate. Maybe you're running Jira this year, maybe you're switching to Asana in a couple of years. Maybe over a 5 year period you're moving from Teradata to Snowflake. Which cloud env do you deploy Snowflake to?

Your negotiating power is then in the momentum of change. If GCP is working better for you than AWS right now, send your new spend to GCP where possible and where stuff is rotating out, move the new stuff to GCP. If you just got a good deal from Azure, start moving in that direction.

This is one of the 'big company vs small company' things, by the way - it doesn't make as much sense for a 100 person startup. But FedEx are a 300k employee juggernaut who spend $75B each year servicing $100B of revenue. They'll have a lot of tech in use.

Re:FedEx to close data centers, retire mainframes by marcosdumay on Tuesday July 05, 2022 @01:23PM (#31991046)
The optimum is very likely near one of the extrema. Either 0% or 100%.

Anything in between will just mix all the problems of a cloud with all the problems of running a datacenter.

Re:FedEx to close data centers, retire mainframes by raxxorraxor on Tuesday July 05, 2022 @10:49AM (#31988863)
I host a lot on AWS but you will need as many people in IT support as before. You can use consultants for a time but after a while your infrastructure will disintegrate because nobody knows about the intrinsic properties the infrastructure of your company.

I believe the self hosted options generally offer vastly more flexibility and can innovate at a faster pace. But only FedEx will know if it is a good decision, perhaps their infrastructure was indeed not competitive. But I heavily doubt they will save 400m in the long run.

AWS support is actually awesome and you usually get responsive within a day, even smaller businesses. But cloud hosting is only useful for certain scenarios. In this case it is Azure and I believe Microsoft currently innovates far slower than Amazon. Of course Amazon could be a direct competitor to FedEx in a few years too given they do more an more logistics. If they aren't a competitor already. In this context I would understand the decision to close their data centers because it is hard to compete with Amazon in the cloud space... a computing alliance with Microsoft would make sense.

Re:FedEx to close data centers, retire mainframes by nemothekid on Tuesday July 05, 2022 @11:04AM (#31989079)
>I host a lot on AWS but you will need as many people in IT support as before

This doesn't pass the smell test at all. Datacenter ops are way more complex than AWS ops; why would someone on cloud need IT support for VMWare, BMC Ops, Power management and generation, supply chain management and land leasing/renting? These are the few roles, off the top of my head, that just go away when moving to the cloud at FedEx's scale.

Re:FedEx to close data centers, retire mainframes by pbalau on Tuesday July 05, 2022 @11:58AM (#31989846)
This is because most people insist on not acknowledging that running your own datacenter will involve "mundane" tasks as "sourcing bolts on a Sunday before Thanks Giving". For a great deal of the people I spoke to, "running your own datacenter" means "we hired 3 servers from a thing in London and we had 1 guy installing linux on them".
Re:FedEx to close data centers, retire mainframes by raxxorraxor on Wednesday July 06, 2022 @03:34AM (#31997903)
Well, I didn't really meant data centers, just in general on premise infrastructure. There is a difference of course. Our administration is almost completely in the cloud, but for production and some adapted applications we still need local data. There is simply no technical solution to run everything in the cloud and we also want to retain some data locally. VMWare is an example. UPS and backups are another. Sure, we use service providers for provisioning of new VMs but IT still needs to do maintenance of the software that runs on it, needs to keep inventory about the servers running in the company. And if you have one VM running additional provisioning is not too much work.

But sure, at the scale of FedEx this is true, they also probably have a very high percentage of administration only. Although I still think the cooperation with MS was still a strategically planned after paying attention to Amazon.

Re:FedEx to close data centers, retire mainframes by nonfamous on Tuesday July 05, 2022 @10:38AM (#31988731)
Probably not AWS, given that Amazon is a major competitor to FedEx in the logistics sector.
Re:FedEx to close data centers, retire mainframes by kragen on Tuesday July 05, 2022 @07:39AM (#31986991)
Boeing jets are a lot more like IBM mainframes than they are like Dell servers, which are still mostly commodity.
Re:FedEx to close data centers, retire mainframes by kragen on Tuesday July 05, 2022 @07:17AM (#31986805)
It's true, of course, that every company is deeply reliant on its web of suppliers and customers, and it is common for a company to have a single supplier or customer that has enormous power over it. But I think you may not appreciate the impact of that situation.

FedEx might have a viable alternative to Boeing for repair parts, but not for planes, and more broadly I think you could make that argument about US airlines in general in the late 20th century: despite things like the MD-80 (before the merger) they really had no alternative to Boeing. It is perhaps not a coincidence that the net profits of US airlines in the late 20th century were almost exactly zero, while Boeing was and is one of the most profitable companies in the world. So far Boeing isn't squeezing FedEx that way, but not because it can't.

(I do note, though, that FedEx owns its own jets rather than renting them.)

But I think there's a different sense in which informatics are more core to FedEx than even flying. FedEx is a corporation, which in its essence is a set of business processes: relationships, practices, and information; unlike a 19th-century railroad, its physical assets like airplanes are relatively expendable by comparison. Historically, those processes happened mostly in people's heads, especially the heads of its managers, but today the vast majority of FedEx's business processes are automated, which means they happen in FedEx's data centers.

And whether those automated business processes happen at all, and how well they happen, is a matter of competence in informatics. Dan Luu argues convincingly that Twitter's kernel team has been key to their ability to execute in https://danluu.com/in-house/; FedEx needs not only data centers but a kernel team and a convex optimization algorithms team.

It's extremely common for companies that are deeply dependent on a single supplier or customer to end up either merged into that other company or bankrupted by the situation.

As for the country, when I set it up I'll be needing yeomanry regiments. You in?

Re:FedEx to close data centers, retire mainframes by beowulfey on Tuesday July 05, 2022 @07:10AM (#31986767)
The better analogy would be that Fedex, since they need to transport things from city to city, could choose to buy jets to transport with instead of renting them or contracting a company to fly for them, which they already do. To remove datacenters is more like removing the jets.

Edit: removed rude phrasing.

Re:FedEx to close data centers, retire mainframes by chrisseaton on Tuesday July 05, 2022 @07:19AM (#31986823)
> The comparison you meant

No that’s not what I wrote.

You don’t buy a jet outright and forget about the supplier - you’re eternally reliant on their service, certification, parts, etc, as regulation is so high.

Re:FedEx to close data centers, retire mainframes by beowulfey on Tuesday July 05, 2022 @08:05AM (#31987178)
Yes but you are comparing to having a datacenter vs removing the datacenter completely. In this example FedEx has removed the datacenter completely, instead relying on someone else’s servers. So the correct comparison is to remove the jets and let someone else fly for you.
Re:FedEx to close data centers, retire mainframes by chrisseaton on Tuesday July 05, 2022 @08:16AM (#31987285)
> So the correct comparison

That may be your comparison but it’s not the one I meant.

Re:FedEx to close data centers, retire mainframes by beowulfey on Tuesday July 05, 2022 @08:30AM (#31987399)
I should have said the more appropriate comparison rather than phrasing it rudely as I did; sorry for that.
Re:FedEx to close data centers, retire mainframes by kragen on Tuesday July 05, 2022 @08:23AM (#31987351)
It is discourteous to claim that someone who says something you disagree with actually intended to say something else.

And believe me, I know a thing or two about being discourteous.

Re:FedEx to close data centers, retire mainframes by beowulfey on Tuesday July 05, 2022 @08:28AM (#31987386)
That’s true, I should not have phrased it like that.
Re:FedEx to close data centers, retire mainframes by spr93 on Thursday July 07, 2022 @12:31AM (#32009925)
Your airplane analogy is apt, but it cuts the other way.

In fact, FedEx likes to buy out Boeing facilities so that they're not "completely reliant" on Boeing for anything.[1]

As for the more general cloud v. on-prem debate, I usually don't like analogies, but the plane analogy is good because jets and logistics-management systems are two things that FedEx needs to handle extremely well to compete.

So, what does FedEx actually do wrt jets?

FedEx both owns and leases them. It mostly owns them outright. Its owned jets are a mix of new-ish to very old (think DC-9s). It also leases and some financing magic (such as some big lease agreements for new 777s a couple years ago).

The reason is that FedEx generally gets excess value from owning rather than leasing, but there are some circumstances where leasing makes sense.

Same goes for "cloud" deployments. The correct answers to cloud "versus" on prem for large organizations like FedEx - "it depends," "that's a false dichotomy," and "it's almost certainly a mix of both" - are neither interesting nor simple, and so those answers don't get execs' attention...or headlines.

For FedEx to brag about taking an extreme position on cloud deployment is breathtakingly foolish. If I were an investor, I'd want to hear something like, "Based on a careful analysis, we've decided to shift certain specific operations to cloud-based systems. We plan to maintain control over the infrastructure and operations of mission-critical systems that have demonstrated resilience." (I'm assuming the latter are systems like the financial institutions rely on, where purpose-built stuff like mainframes using IMS have demonstrated near-zero downtime and darn-near-bug-free software for decades, because FedEx, like banks, needs to handle lots of simultaneous transactions in real time without error or deadlock.)

[1] Example: https://www.ch-aviation.com/portal/news/102874-fedex-to-take...

Re:FedEx to close data centers, retire mainframes by kragen on Sunday July 10, 2022 @11:39AM (#32045710)
This is astounding; I had no idea. Thank you for explaining the details. Do you have publicly citable sources for the information on the owning vs. leasing?
Re:FedEx to close data centers, retire mainframes by yobbo on Tuesday July 05, 2022 @12:04PM (#31989955)
> For example FedEx flies Boeing jets

This, in particular, is a bad argument since jets are replaceable and interchangeable.

Possibly, the only thing that is not replaceable in FedEx is their information system.