At the same time, you need mindshare at the lower end to drive demand for the high end. I previously developed for POWER by getting an Apple Mac Mini. Very affordable, and while not a full POWER instruction set was sufficient to test and validate my code. Today, there is no equivalent, and I'm about to dispose of the Mini. I couldn't justify use of POWER today even though we have a datacentre stuffed full of machines for HPC, and could likely make use of it. We have zero experience with it, and no easy way to get any experience. Being able to buy a cheap low-end system for experimentation is important. If I wanted to do this for ARM, I could buy half a dozen cheap boards online this morning and have them delivered by Monday. But POWER is out of reach.

Except for special US DoD things (like supercomputers that simulate nuclear weapons) for Sandia Labs and the like, the math also doesn't work out for HPC...

Let's say that a 44RU rack of 1U, narrow (half of 17.5" width), dual motherboard, single socket systems costs $N if it's built with Xeon or EPYC CPUs. As an example here the Supermicro barebones which are two long, narrow motherboards in 1RU with single sockets and front to rear wind tunnel airflow. Same rack of systems costs $N multiplied by 2.25 if built with power9, but only performs 1.25 times faster.

I don't follow the logic. Where are you getting your pricing information from? Why do you think power is more expensive?

the cheapest power8 server I have ever seen was probably 4x more expensive then the equivalent dual socket xeon. That info is a bit stale. But there is also a complete lack of good information online about actual dollar figures for power8 systems (a lot of "contact us for more info!"). The platform's pricing is aimed at the sort of CTO people who don't care about the cost of Oracle licensing. Not the sort of people who care about $/MIPS for HPC applications or cloud scale hypervisor platforms.

I would very much like to be proven wrong so if you have access to some specific "this hardware spec costs this much money and you can buy it now" info, that i can compare to barebones supermicros, please do share it.

You're right in that the sticker price is much, much higher. I'm surprised you were even available to find one. IBM tends to like to keep things secret until you reach out to them, for better or for worse. I would like to see some public power 9 benchmarks to compare to xeons instead of just marketing speak as well.

But I will say that your comment about $/MIPS isn't accurate. There are many people, including Oakridge national labs, who are using it because of the higher performance and v100 integration. Intel is obviously not too keen on letting Nvidia succeed.

somehow, compared to the vast amounts of money they spend on other things, I think pricing is way down the list of things that are important to organizations like LLNL, Argonne, Sandia, Idaho National Labs, PNNL, etc. Basically everything that uses HPC for DoD top secret projects.

Email supermicro and ask about pricing, they are trying to gauge interest currently.