An existing environment is usually a large pile of twisty dependencies on operating system calls, on database or file formats that aren't entirely portable, on language extensions that don't exist elsewhere, on objects built from source code that's been lost, on dependencies on products from vendors that no longer exist, and it's all often inextricably linked with front-ends that would have to be rewritten, data bridges that would have to be rewritten, and users that need to be retrained.
In some cases, this effort can involve re-architecting substantial parts of a manufacturing facility that's involving six or ten buildings and that are each most of a kilometer long. And with many industrial-scale environments, change is not going to happen. You (might) see a new environment implemented when the entire production line is nuked and paved at the end of its useful lifetime, and replaced with a newly-deployed production line. In other cases, you'll probably see the existing system patched and re-used.
For many of these cases, just ramping up a load test would be a substantial effort.
Some hunks of the environment will have test cases and specs and documentation, and some hunks probably won't.
For many of these installations, substantial downtime has SEC-level visibility, or can involve exposures to various legal entanglements, or can involve re-establishing politically- or legall-mandated external certifications, or can cost thousands or millions of dollars per unit time off-line, or requires the production line to be cleaned with a jack-hammer or entirely replaced, or other issues.
If you can afford an incremental port or a big-bang port, well, what's that cost in comparison with the incremental costs of keeping the existing environment going?
C-level folks just don't blindly sign the really big bills involved in keeping these environments going, and most competing vendors seeking to acquire least part of these mainframe-scale budgets.
Sure, you could move those apps to a bunch of smaller boxes. Then you open yourself up to server failure, admin costs of maintaining multiple boxes, etc.
Mainframes wrote the book on SaaS long before 'net cloud computing came along. IBM and others have operated computing as a utility for years and years. What manager who gets his IT done on rented computer time is going to suggest taking the apps in-house on a bunch of servers? None that I know of.
It's pointless to invoke the almighty Google; they simply don't deal with problems in this class.
FWIW, I worked at a major bank (top 5 in the U.S.) and listened in on several discussions between their top tech guys. As mentioned in other comments, their number one worry was the supply of COBOL coders.
They were terrified of Google and/or Paypal and what would happen if one of those companies got serious about competing in the banking arena. Most of the top architects _knew_ that it was possible to build a bank on a huge cluster of servers. They also knew that there was no way an existing bank could build the infrastructure and software while also maintaining their existing infrastructure.
Even then, there are consistent, synchronous databases that run on Linux clusters. Oracle RAC and Sybase ASE come to mind. Linux also supports hardware NUMA implementations, which allow single system image software. An emulation layer could run on top of NUMA hardware.
PS: I suspect a large part of this is the mindset and experience of the people building the software. Well written Java code can be about as fast a well written COBOL code, but the average Java system is horribly inefficient.
Edit: An 1980 IBM 3081 could access up to 32 megabytes of main memory. A 5400-series "Harpertown" can have 24 megabytes of cache. So Modern CPU cache memory vs Mainframe main memory is not an unfair comparison. But that's just overkill the real shocker is 3081 had less bandwidth to it's registers than many modern computers have to it's local network. (It was below 40MHz and consumed 23 kilowatts of power).
PS: Feel free to post some actual numbers.
Mainframes from IBM are under constant revision and update, just like server tech.
PS: We live in an age of horrable custom software, sitting on a monstrous stack of terrible commodity software. If you spend a noticeable amount of time waiting on a modern computer system it's really just a software problem. (Wait you actually want to use XML?) But, often upgrading the hardware is really the best option.
The mainframe would suck for what you'd use the Altix for, e.g. weather forecasting. The Altix has a lot of processes that want to work on a common dataset, but it would grind to a halt if you tried to use its interconnect to serialize transactions.
Do mainframes have some kind of crazy low-latency or high bandwidth bus that something like the Altix is missing? It appears they use similar hardware architectures, at least from a high level: super high speed modular backplanes with blade servers that form a single system image. The NUMALink 5 is stated as having 15GB/s of bandwidth (7.5GB/s in each direction) with a ~1µsec latency.
Or we can just say that all languages that are Turing complete are equivalent...
BTW, Google needs with MapReduce are fundamentally different - it favors scalability and thoughput over correcteness vs banks that need distributed transaction consistency most of all.