For IO intensive transaction processing with high security and low downtime requirements these systems are quite sweet.

IBM sells it with moniker "Pervasive Encryption" but if you look under the hood, it is actually impressive and very detailed.

Build in security at the hardware level. Especially the Crypto Express6S card. There is hardware acceleration for everything cryptography (including SHA-3 and SHAKE), tamper sensing, true random number generation. Encrypted paging in z/vm, etc. SSL/TLS can run fast. Security auditing is not an afterthought.

Of course you can add all this in special cards for x86 server racks but the cost is probably the same with lower IO throughput and less integration. Getting the same uptime in production would be significantly more expensive if not impossible.

How does it differ from using third part HSMs with commodity hardware?

last paragraph I wrote.

On unreliable commodity hardware, you would build a cluster and cluster OS / scheduler that would keep your service and jobs running while individual servers died. With a mainframe, you save costs on software development by just assuming the hardware is 100% reliable.

The z14 has a ton of cache and does everything it can for single threaded performance, but for the money you could buy a ton of cores (Intel Xeons or AMD chips) and trounce the mainframe on benchmarks that can use the parallelism.

For the money, you would be able to buy the gear to have a lot of bandwidth between nodes and to the storage.

My guess is the mainframe would lose the CPU benchmarks and win the IO benchmarks.

That, and trounce the rack in uptime by a couple orders of magnitude.

Single-thread performance is also pretty high, so it'd only lose in the aggregate CPU benchmarks.

"trounce uptime by a couple of orders of magnitude"?

Not true. If you view the network as the computer, then the network is now forever winning.

What is Google's or Facebooks uptime? Their computer is their cluster. Those clusters are never going to be powered off ever. They can upgrade and replace the entire cluster without being down. If you have a cluster of 10,000 nodes, turning off 100 or 1000 is not downtime, so longer as it's up and serving. No mainframe ever will run for the next 100 years. It's a possibility that a cluster built today could be running for the next 100yrs.

> What is Google's or Facebooks uptime?

99.9 percent. Cloud services from Microsoft, Google, Amazon typically have only three nines or less for paying customers. They also seem to have same level of uptime in their own services.

But these are not "firm guarantees". For example Amazon just gives service credits after downtime exceeds what is promised. They "undergo unscheduled outages" from time to time. Whole regions are down at once.

Remember Amazon's Prime Day outages? It's not good for business when tings go down.

For workloads where eventually consistent is okay.

It's harder to get mainframe like uptimes for OLTP simultaneously with mainframe like IOPS.

Banking uses eventual consistency all over the place. Unsurprising given they started with letters going by ship and courier.

Exactly, I think that folks forget that ATMs operate like that, if the network is down ATM doesn't stop working. That's why folk's account can go below $0. Banks have been operating distributed systems with eventual consistency for a very long time.

Three 9s vs Five 9s. The mainframe wins here. Heck I remember moving a system for a bank that had been up for 23 years.

the scope of the comparison was to a single rack, not all of Google or Facebook.

And even then, what is their non-degraded uptime? There are parts of their infrastructure failing all the time being routed around by incredibly clever software.

And I bet all that costs much more than an IBM. Not a computer, the company.

Anyone know what the biggest single-system-image x86 system on the market today is?

One of my co-workers manages the SGI UV300 described at http://med.stanford.edu/gbsc/uv300.html (new purchased get hardware badged as the HPE MC990 X). It does fit most definitions of a supercomputer, as it's multiple nodes with interconnects that allow a single OS instance to run across all of them (the OS then sees & manages the separate NUMA nodes).

One thing that has been learned is, software will sometimes do interesting things if presented with so much memory. For example: https://github.com/zfsonlinux/zfs/issues/7275

The group I'm in at NASA Goddard has two UV300s. They have 96 cores each (no hyperthreading) and 6TB of RAM. And they are loud.

The SGI Altix line, which used Itanium 2, could go up to 1024 cores with a single system image, which is just insane. We had one of those at Goddard a decade or so ago, but I didn't have access to it.

I'm fairly sure UV (or some SGI labelling) has gone up to 1024-ish cores in a single system image, but I don't have a reference. (SGI Origin 3000 was spec'ed to 512 cores long ago.)

1536 cores on Cambridge’s COSMOS SGI Altix UV2000 system:

http://www.cosmos.damtp.cam.ac.uk

Speaking of SGI, I was expecting the Cray XC family to be able to do that with the ludicrously fast interconnects they have, but I couldn't find any mention of it.

No, an XC40 runs a kernel on each node, the programming model is MPI, not SSI with NUMA. It runs a minimalist Linux distro called CLE. The nodes are surprisingly small, in terms of memory too, 64 or 128G. Physically they are beautiful to look at, so clean, just a CPU and some memory and nothing extraneous. Cray really should do glass cases.

They used to make beautiful computers.

The salient hardware difference is probably cache coherence in the SGI interconnect.

No, the Ares interconnect is brutally fast but it fundamentally isn’t doing anything as complicated as that. It acts as both storage and networking between the nodes.

I missed this at the time, but yes -- that's the point, in comparison with NUMAlink or whatever it is now. (s/Ares/Aries/)

Maybe the HP Enterprise's Superdome line (at least some of the tech is formerly SGI UV series), up to 32 Xeon CPUs / 48 TB RAM? https://www.hpe.com/us/en/servers/superdome.html

Probably very similar to https://www.huawei.com/en/press-events/news/2016/3/Worlds-Fi...

The biggest I heard of was built by SGI (although it was just the husk of SGI which was was bought by another company which kept the name), now HPE. It had up to 32 Xeon sockets (so maybe 256 cores?) and up 24 TB of RAM.

"Similarly priced" means that's a lot of x86 servers.

You can get 8-way Xeon Platinum with tens of terabytes of memory that'll more closely mimic a mainframe. You probably won't be able to fit more than 3 in a rack and they will be hideously expensive (but, at least, will have hot-swappable storage and power supplies) and will be mostly air-cooled.