Having never worked directly with a mainframe, I can't really speak to the reliability question. What I do know, though, is that I've spent my entire life living around PC hardware, working on PC hardware, etc, because that's the hardware that's designed to fit into my day-to-day life. So, if I have a new idea, and want to try it out, I'm going to reach for the hardware I already have, because it's basically free. Just like Mark Zuckerberg did.
And at that point, I'm already on a path that leads inexorably toward building the reliability into the software. There's simply never going to be any point at which it makes sense for me to scrap everything I have so far and do a complete rewrite for a different platform.
Absolutely agreed, "path dependency" is a big issue that comes up in a lot of different contexts both in the natural world and human endeavors. Just because from a high level or after the fact someone can identify a much more optimal final result doesn't mean that result was actually realistic to get to, or would be worth moving to from whatever local minima a project ended up in. I think path dependency plays a major part in what makes "disruption" possible, and in how businesses can sometimes be eaten from the bottom. A lot of companies end up in a state where they have a bunch of end point products but they haven't considered the paths necessary for users to reach that end point. Without a ramp their pipe can slowly empty.
Yet people still often claim that it makes sense to go to all the time and trouble and expense of moving off of reliable legacy platforms onto other platforms which aren't naturally anywhere near as reliable. Imagine that!
But that was only introduced in 2010, so either the timeframe I was guessing at is wrong, or I'm misremembering what component it was. Could have been disk drives maybe.
Yes, I've seen this too, and on lowish-end servers.
Doesn't help when your mainframe's infiniband controllers shit the bed and take the whole box down (a thing which has happened to me).
In any case, it helps explain why a purse-string-holding exec would be convinced to shell out money to IBM.
I wasn't really meaning to say that a Spark cluster is as robust as a mainframe (though maybe someone's figured out some tricks), more that I could totally see some sales folks conducting the same demo using something like Spark.
You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.
Meanwhile, these days you have folks running "start of the art" systems which may just roll over and die for no discernible reason. These may not easily come back up, either, if at all. That's why they have to have so many of them!
I was in college at the time and looked at job postings for mainframe programmers because I wanted to work on that. Never found one that required less than 5-10 years experience, not then and not when I've checked every couple years after that. Too bad; I quite liked the idea of working in an ecosystem that starts from "let's make this work every single time" instead of "let's make this work well enough to keep customer complaints to a dull roar".
I've been part of a team that ran Linux on s390x, and it was a great exercise in demonstrating that however reliable your mainframe, it's still a SPOF. Even if you don't have my bad luck of four catastrophic hardware failures in as many years, your one reliable box still depends on the power, network fabric, storage fabric, etc at a single site. If you want to avoid losing your business when a sparky screws up, you need... several mainframes. At which point you're spending an absolutely astronomical amount to either sysplex, or you're building resilience into your applications, just like a bitty box.
And yes, so many mainframe folks are so very out of touch with the broader world. In around 2016 I had the supposed regional tech expert on z-Series system lecture me (rather snidely) on how:
1. s390 processors were 20 - 40 times more performant that Intel processors (demonstrably not true for any workload I cared about). 2. You could not do virtualisation on x86_64. It was impossible. With a straight face, in the year 2016, this guy told me, and clearly believed, that it was not possible to run heterogenous virtualised workloads on an Intel processor. Apparently tens of billions of Jeff Bezos' net worth literally did not exist for him.
There was more, but it was like talking to someone who had been frozen in a block of ice since 1996.
Always remember (and don't ever forget) that the only reason most of us have even heard of Intel and Microsoft is because of their relationship to IBM back in the day. They weren't chosen, nor did they ascend to great heights, based the quality of their products. Rather IBM chose them mostly for the sake of its own convenience, and then they leveraged that relationship to the hilt.
BTW, just because a speculative execution attack or whatever is theoretically possible on an IBM system doesn't necessarily mean that it can be carried out in any practical sense, given the overall design of those systems and their general level of built-in security. But IBM can't just sit there and ignore the possibility, either.
Another problem they have these days relates to open source software and such, which they've been porting to their platforms much more lately. If a security patch for that software comes out then they still have to apply it, even if there may be no practical way to exploit it on their systems. And it can be quite unnerving to see long lists of such patches show up on a regular basis for systems which are otherwise generally considered to be rock solid.
As for Facebook and Google and Netflix and so on, you have to remember that for all their claims of reliability and such, their stuff really only needs to be "good enough". But good enough generally just doesn't cut it when dealing with things like financials and such.
BTW, I have a colleague who works for a massive corporation - one that is still recovering from a ransomware attack which has so far cost them at least $300 million. They have thousands of servers, and she says that it has now become IT's full-time job just to keep those updated and patched. (There's apparently little or no time these days for silly little things like development and testing and code loads and such.) I didn't ask her if they are still planning to replace their few remaining mainframes (I'm guessing they're in no big hurry now), but she did tell me that upper management currently thinks that "The Cloud" may yet be their road to salvation.
My startup was acquired by BMC software, one of the largest third party providers of mainframe software, and I also worked for EMC for a time early in my career, so I spent a lot of time hanging around the mainframe space. Two quick anecdotes:
- working on a giant data center move for a big bank, there was a break room filled on a Saturday with operations people from the “open systems” (non-mainframe) and mainframe teams. I was struck by the contrast: the Unix/windows operations folks were tattooed, pierced, jeans and t-shirts, late 20s. The mainframe folks were short hair, polos and jeans, mid-50s. The two groups self segregated into clusters on opposite ends of the room, not speaking to each other.
- at a BMC leadership offsite, I found myself st the dinner table with a couple of mainframe engineering and product directors. After a few drinks, we got on the subject of why more tech companies weren’t using mainframes. “They’re so reliable! Why wouldn’t you just go out and rent one from IBM, and then you don’t have to worry about uptime or stability?”. I explained that the fashion had become to build the reliability into the application and assume that the hardware was not reliable. This confused them. “That sounds like a huge pain in the ass, why would you bother dealing with that?” So I explained that the cost per compute cycle was so much cheaper that it made it worth it, plus open source, etc etc
One of them kept pressing: “but Facebook could just take all the engineers they’re devoting to building reliable infrastructure and shift those people to writing customer facing code!”
The other director stopped him and said, “don’t you see? They’ve already done it. There’s no reason for them to go back now. People have figured out that they don’t need mainframes to get mainframe reliability”. The other director just kept shaking his head and we moved to other subjects.
This conversation happened in 2014.