A not-so-modern but fun anecdote:

My father worked on a product called SnapVantage in the late 90s and early 2000s, which basically used the copy-on-write abilities of the virtualized mainframe disk in a StorageTek disk array combined with z/Linux and z/VM as a hypervisor to instantly provision and boot Linux machines.

It looked like a very, very toned down version of the AWS management console, with a near-identical ability to select a distribution, upload SSH keys, find an IP, and have a full, running VM in seconds... but this was in 1999, when watching a Linux box "copy," provision, and come up in just a few seconds was mind-blowing.

I used my nascent Perl skills (as a ten-year-old) to write the installer for the host software, which I believe actually shipped to customers nearly unmodified!

The zSeries still supports a lot of battle-hardened HA features like lock-step execution, hot CPU fail-over, and hot-swap memory and IO which are very rare, esoteric, and hence hard to support in the x86 world (and rare, esoteric, and untested is not what you want in your HA system).

Plus, the number of VMs that can run well on a z/Linux and z/VM environment is amazing - we had hundreds and hundreds of instances running on one zSeries.

There's a very full-featured zSeries emulator called Hercules which will allow you to run z/OS and z/VM (if you can find them, of course). There's also a distribution of MVS 3.8j, the 1980s System 370 / System 390 precursor to z/OS, available for Hercules for free if you want to get a taste of what administering such a system is like. A lot of obscure job-scripting languages like JCL and early scripting languages like REXX were developed for early mainframes and still live on today.

Ex-IBMer here. I worked with WebSphere (IBM's Java App Server) on zOS, and am by no means a Z-series expert.

My understanding is that these machines are amazing at IO, and handle virtualized environments incredibly well (and have so for decades) (e.g. thousands of Linux machines running).

If you need raw CPU performance then a mainframe is not for you. If you need competitive pricing then a mainframe is not for you. If you have a mainframe and need a have an expiring agreement then this machine may be for you. If you have legacy mainframe code then this machine may be for you.

I remember watching a financial update from Sam Palmisano (IBM's previous CEO). I swear he started to drool when referring to IBM's mainframe margins. From my experience mainframes are for legacy companies with legacy systems, as they are not price competitive with other markets that experience real market competition. But, it is interesting and incredibly reliable hardware.

> My understanding is that these machines are amazing at IO

What makes them amazing at IO?

Isn't IO more a result of what type of disks a computer has? An SAS 15k disk can only do 300 iops whether it's a mainframe or a commodity server, no?

I/O is about channels. Channels are fixed bandwidth queues which generally have a 'signaling' rate, and a 'bandwidth' the one that most people are familiar with is a SATA port.

A SATA port can do some number of IOs per second (this depends on how the controller is designed for the most part) and the SATA data pipe is 6 gigabits/second or 600 megabytes per second these days.

A SATA Disk (the spinning rust kind) is different, it can pump out some maximum amount of data per second (usually about 100 Mbytes/sec and has an average seek time of a few milleseconds. So if you are doing random i/os to the disk, so that every I/O needs a seek, and the seek takes 10 mS, you can do at most 100 I/O operations per second (IOPS) if you transfer 512 bytes with each IOP you get a throughput of 100* .5K or 50K bytes per second. If you transfer 100Kbytes per transaction you get 10 megabytes per second. Anyway the bottom line is that an I/O channel is bound both by its bandwidth and the number of IOPS it can do.

A PCIe 'lane' can do 250Mbytes per second and with Intel's 5500 chipset about 2 million IOPs. (Although those IOPs are spread across all PCIe channels) A "big" server chip set will give you 32 or maybe 48 PCIe "lanes" which can be 12 by 4 (or x4) lanes or 2x16 + 8x1, or 2x16, 1x4 and 4x1 (this is a common desktop config for two video cards and misc peripherals)

The bottle neck is exactly like networking, each I/O device can consume a certain amount of channel bandwidth, since they don't run 24/7 you can "oversubscribe" which is to say potentially have more I/O devices than you have bandwidth to service them (this happens a lot in like big disk arrays) and ultimately the transaction rate of the entire system is constrained by the cross sectional bandwidth of memory and I/O channels such that you get I/O limited in terms of things you can push around.

Anyway, a mainframe has thousands of channels. Lots and lots of them. And each of those channels has a lot of raw bandwidth and can sustain a high transaction rate. This makes a mainframe a transaction monster.

In the taxonomy of computer systems, 'supercomputers' have a monster amount of memory bandwidth and high transaction rates (low latency) between their compute nodes (or cores). Mainframes have a monster amount of I/O bandwidth with high transaction rates between their nodes and I/O devices. General purpose computers achieve good economics by limiting memory and I/O bandwidth to stuff that will fit in a single piece of silicon. (or at most a couple of pieces)

Thanks for the well-thought-out and detailed explanation.

Mainframes tend to have prodigious amounts of expensive, unusually fast, specialised RAM. And then they have extremely smart controller hardware that the OSes can hand off almost everything off to.

And this is a theme replicated throughout mainframe design. Everything is replicated, all pathways are wide as possible, everything is integrated from the hardware up to the application layer.

Take virtualisation, for instance. IBM mainframes have PR/SM, a hardware-level capability to virtualise everything in the system down to the lowest level of granularity. And they've done this since 1985.

Plenty of other machines have Amazing IO in this day and age now. It is a fairly old notion now that Mainframes were much better at IO than other machines (this /was/ true).

That said, they do deal with IO pretty well. They have dedicated offload processors for transferring data, which means for the same (IO intensive) workload, the CP utilisation of a zSeries machine would be much less than that of an x86, pSeries (or other) machine. And in terms of the disk being a bottle neck... If you're pushing to a disk array that has 192GB (pretty standard) of write-back cache... then no, the disks are not a large concern.

But... At the end of the day, does this really weigh up to the cost of Big Iron, or for that matter the additional licensing of software on top of it?... Not really.. is my answer.

Back in the day mainframes had an I/O advantage because they offloaded a lot of I/O processing to coprocessors (e.g. a minicomputer would just have one processor while a mainframe would have one main processor and one or more I/O coprocessors). That advantage appears to be gone now because I think they are using normal PCIe cards and normal cores as I/O coprocessors (which you could also do on a non-mainframe server).

Of course, mainframes are also faster simply because they are more expensive (e.g. 100 PCIe slots are obviously faster than 10 slots), but that's not an intrinsic advantage (in theory you could develop a $10M x86 server — call SGI if you're interested).

Think throughput...

Lots of cache memory and very fast buses to very large external disk arrays. An old mainframe trick was to use the previous generation hardware as the IO controller for the current generation...

I used to develop software for system-z(2 years ago). As someone else said, if you are already using mainframes, get another one. If you arent, pretty much you can ignore them. Their hardware is of higher quality, but I think their uptime has more to do with the hermetic seal of human process that exists around the installations. You can't get anything changed in production, ever. And thats fine, because banks have been doing the same thing for the last 30 years.

I don't actually think performance wise they have anything on a cluster of x86 servers. But its hard to benchmark things like that accurately.

From a developer point of view, its not a great technology stack to work on. Its old and out dated, and is pretty incompatible with other platforms. EBCDIC is a pain, 3270 terminal is a pain. Having to allocate files manually is a pain. Having actually about 7 different types of files is a pain.

Its really all a pain. Just dont go there.

Also, I found that guys who had been working on mainframes for a really long time have a pretty outdated view of what the "distributed" world has to offer. A lot of the time they dont understand what "web Scale" at the level of google, facebook et al are operating at.

As an additional funny annecdote, the I/O that a lot of people have been talking about is now conventional if high end disk array's AFAIK. But internally the disks are emulated for the OS to look like the disks it was using back in like 1975. Thats how it gets backwards compatibility, just dont change anything.

As a technologist I'm always excited about new stuff, but there's another part of me that thinks it's really neat that some of the world's most important software was written 30 - 50 years ago and still works just fine.

Can you imagine a time 1000 years into the future where software written today might still be in use? How cool would that be?!

Most of that software is still in use because it's very poorly written using a combination of COBOL and assembler and people are afraid to touch it.

Very poorly written and it works, at least.

I'm currently reading a book written in the 1170s, so I can imagine that :)

And I've just finished the Iliad (in translation) which dates from about 800BC describing events of ~1250BC.

It's a bit like the descendants thing that was discussed a few days ago on here - either lots of stuff will survive the next 1000 years (and therefore lots of running code will) or almost nothing will.

There are some technologies out there in the non-mainframe world that have similar properties.

To name the ones I know of - vi, emacs, C, Common Lisp. These have held pretty steady for about 30 years now.

Given I'm sure many people here have anecdotes about little snippets they wrote in a short amount of time that ended up in use for a decades, even though the code was crappy (someone tell me I'm not alone here...) - I'd be willing to bet something, somewhere will still be around a long time from now.. though a thousand years is a bit hard to comprehend in terms of technological change.

If banks still exist they'll still be emulating zOS on their quantum dimension-folded computers...

Cool? I'd think that would be depressing.

>Their hardware is of higher quality, but I think their uptime has more to do with the hermetic seal of human process that exists around the installations. You can't get anything changed in production, ever. And thats fine, because banks have been doing the same thing for the last 30 years.

You hit it. Things break when you change things. The high priesthood that surrounds all mainframes prevents that from happening.

Nowadays, if you have to change stuff, IBM sells you AIX boxes that cost nearly as much as mainframes which sort of integrate with them to support those changes.

A friend of mine worked for several years as a dinosaur herder and still does on-call work for his employer's data centre.

Basically it's a different universe. Everything that might be familiar to someone with a unix or windows background is ... different. More eldritch. More powerful and yet more obtuse.

Oh and JCL. JCL is the devil's work.

One thing they did that I thought was cool was to consolidate a bunch of Linux servers onto their mainframes. They replaced a few racks with z/VM instances. If they needed another server, they could spin it up in literally seconds if the template was already loaded. This was back in 2004, 2005.

You don't buy (actually: rent) a mainframe (actually: CPU cycles and RAM capacity) because it's cost-effective. You do so because the system absolutely must work and you have a mind-numbing number of transactions to process in a hurry.

If you need it done in a hurry, you don't use a mainframe. Their performance is actually quite poor compared to pretty much anything else, as is the cost per transaction. In general, if you need a machine where the hardware never fails, and you can't get the level you need with a cluster, you use a mainframe. Of course, it's still human error that bites you in the ass every time anyway. These days, you really only use a mainframe if that's all you have.

By "pretty much anything else" you mean what, exactly?

My father has worked for IBM, Airlines and Credit Card companies developing on their mainframes. I believe the the system is called z/TPF?

I can't recall any specific stats or stories right now but you can imagine the discussions between a father who develops primarily with Assembly and a son (me) who ended up being a .NET developer working on various startups. We've had a few talks about the 'cloud' mostly resulting in him laughing at what we call 'web scale'.

I had a number of mainframe mentors back in the 90s who taught me about TPF...

TPF is an OS, transaction processing & database system that was written in mainframe assembler and deployed in 1979 to run SABRE and a few other core transaction systems (VISA, AMEX, Marriott hotels, etc.) - maybe 100 customers still use it today - and it was a descendent of a previous assembler system developed in the 1960's.

It probably was (and is) the closest system to "web scale" through the 80's and 90s's in that it ran some of the highest volume workloads globally (e.g. 5000-10000+ tps back in the 80s / 90s, 25000+ tps today) at 99.99% availability. In context, Twitter hits 10-15k tweets per second during big sports events.

These days it's extremely expensive and slow to make changes to TPF relative to modern technology (as it traditionally required S/370 or S/390 assembler, though these days requires C). IBM has a newer version called z/TPF which makes it easier to code in C or C++ and communicate with the outside world via web services, etc.

A very different world...

Thank you for this! :)

maybe 100 customers still use it today

Indeed it does seem like there are only a few companies that use this stuff. Like you said, you had to be a pretty big company with massive amounts of transactions to merit it. He's actually contracting with the IRS right working on some newer systems to handle your tax returns. (He tells me every year it's a miracle you get your tax returns every year on the old system)

Whenever he's looking for a new gig he basically has a set list of contacts he goes through. It seems like almost all the developers of these systems know each other.

Let me say that "web scale" to a startup is a different beast than "web scale" to big companies.

I remember being at a startup and being impressed that we were doing a million dynamic pages per hour. Then I got the chance to see a real system at a big company pumping out orders of magnitude more traffic than that per second. (I am not at liberty to share exact numbers.) And doing constant processing on that traffic.

I doubt your father would laugh at systems like that if he learned what they manage to handle.

Oh I'm sure! His examples usually revolved around his days working on credit card processing systems. But I can't even start to imagine the systems required to handle requests for VISA which can supply the needed response-time and up-time demands.

A hundred million pages per second? Really?

Stop and think about how much traffic you think that Google gets. Don't forget all of the AJAX requests that are sent back and forth, random ads on third party sites (many of whose pages auto-refresh while they are sitting in your browser, each time loading multiple ads), etc.

Let's just say that any semi-reasonable back of the envelope will give you some truly impressive numbers. I don't think that I should say any more than that.

Now consider that this data is constantly being processed to figure out things like click through rates, trending requests, and much, much more. And that data is then fed back into the live site.

The implied transaction rate should be enough to impress the average mainframe user.

Of course Google's scale is impressive but you're talking about datacenters upon datacenters of servers. I always get a chuckle when people talk about "old" mainframes taking up a whole datacenter. Now the Mainframe occupies a corner and the commodity servers take up the datacenter. :P

Facebook allegedly pushes two million images per second.

The Wikipedia article [1] is a pretty good summary. The main advantages are basically very high IO throughput, very good fault tolerance, and very good backwards compatibility.

   [1] https://en.wikipedia.org/wiki/Mainframe_computer#Characteristics

They're wicked cool in every way except the cost.

This one is worth listening to, for learning a bit about mainframes: http://www.se-radio.net/2012/03/episode-184-the-mainframe-wi...

The mainframe scheduler is fantastic. z/OS has the capability of limiting the resource usages in different dimensions, including memory use. You can give priorities to processes, which means that you can define what is business critical and what is not. If the workload goes too high for the machine, then low priority work just waits.

In any case, I'd advise anyone who can have a try at mainframe to give a look, even if just because it's one of the still used non-Unix based systems.

The trend at the moment is having java based web applications that interface with a mainframe. So we have a web front end that interfaces with a mainframe that processes/stores the data.

Most companies are trying to stop using the mainframe so much, it is laughable though because they are moving from mainframe based solutions, to mainframe based solutions.

When I was working for IBM (software group) I heard anecdotes that in case of accident power outage mainframes can continue execution exactly from the same CPU instruction where it stopped and absolutely no data is lost. I don't know if it's true or not but this sounds cool.

I think the best thing is the capacity plan for, say, unexpected trading volumes is as simple as "Ring IBM and get more capacity unlocked."