News for hackers. Stuff that matters.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by owlbite on Friday October 06, 2023 @11:07AM (#37791784)
It seems odd to focus on the CPU here, when presumably the vast majority of those flops re coming from the NVIDIA parts?

Are the CPUs expected to contribute significant compute, as opposed to marshaling data in/out of the real compute units?

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by jefft255 on Friday October 06, 2023 @11:28AM (#37792085)
Yes, CPUs are still the main workhorse for many scientific workloads. Sometimes just because the code hasn’t been ported, sometimes because it’s just not something that a GPU can do well.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by londons_explore on Friday October 06, 2023 @02:44PM (#37794575)
> just because the code hasn’t been ported,

Seems stupid to use millions of dollars of supercomputer time just because you can't be bothered to get a few phd students to spend a few months rewriting in CUDA...

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by bee_rider on Friday October 06, 2023 @03:41PM (#37795292)
>> just because the code hasn’t been ported, sometimes because it’s just not something that a GPU can do well.

> Seems stupid to use millions of dollars of supercomputer time just because you can't be bothered to get a few phd students to spend a few months rewriting in CUDA...

Rewriting code in CUDA won’t magically make workloads well suited to GPGPU.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by wang_li on Friday October 06, 2023 @04:16PM (#37795764)
It's highly likely that a workload that is suitable to run on hundreds of disparate computers with thousands of CPU cores is going to be equally well suited for running on tens of thousands of GPU compute threads.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by atq2119 on Friday October 06, 2023 @05:38PM (#37796778)
Not necessarily. GPUs simply aren't optimized around branch-heavy or pointer-chasey code. If that describes the inner loop of your workload, it just doesn't matter how well you can parallelize it at a higher level, CPU cores are going to be better than GPU cores at it.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by monocasa on Friday October 06, 2023 @05:45PM (#37796845)
They're not that disparate; the workloads are normally very dependent on the low latency interconnect of most supercomputers.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by mlyle on Friday October 06, 2023 @03:24PM (#37795082)
A supercomputer might cost $200M and use $6M of electricity per year.

Amortizing the supercomputer over 5 years, a 12 hour job on that supercomputer may cost $63k.

If you want it cheaper, your choices are:

A) run on the supercomputer as-is, and get your answer in 12 hours (+ scheduling time based on priority)

B) run on a cheaper computer for longer-- an already-amortized supercomputer, or non-supercomputing resources (pay calendar time to save cost)

C) try to optimize the code (pay human time and calendar time to save cost) -- how much you benefit depends upon labor cost, performance uplift, and how much calendar time matters.

Not all kinds of problems get much uplift from CUDA, anyways.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by jonwachob91 on Friday October 06, 2023 @04:16PM (#37795777)
>> A supercomputer might cost $200M and use $6M of electricity per year.

I'm curious, what university has a $200MM super computer?

I know governments have numerous Supercomputers that blow past $200MM in build price, but what universities do?

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by sophacles on Friday October 06, 2023 @05:38PM (#37796771)
University of Illinois had Blue Waters ($200+MM, built in ~2012, decomissioned in the last couple years).

https://www.ncsa.illinois.edu/research/project-highlights/bl...

https://en.wikipedia.org/wiki/Blue_Waters

They have always had a lot of big compute around.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by mlyle on Friday October 06, 2023 @05:46PM (#37796851)
> I know governments have numerous Supercomputers that blow past $200MM in build price, but what universities do?

Even when individual universities don't-- governments have supercomputing centers that universities are a primary user of and often charge back value of computing time to the university or it is a separate item that is competitively granted.

Here we're talking about Jupiter, which is a ~$300M supercomputer where research universities will be a primary user.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by otabdeveloper4 on Friday October 06, 2023 @03:24PM (#37795084)
CUDA is buggy proprietary shit that doesn't work half the time or segfaults with compiler errors.

Basically, unless you have a very specific workload that NVidia has specifically tested, I wouldn't bother with it.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by brnt on Friday October 06, 2023 @03:32PM (#37795157)
The JSC employs a good number of people doing exactly this.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by cmdrk on Friday October 06, 2023 @02:53PM (#37794695)
sometimes the code is deeply complex stuff that has accumulated for over 30 years. to _just_ rewrite it in CUDA can be a massive undertaking that could easily produce subtly incorrect results that end up in papers could propagate far into the future by way of citations etc
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by londons_explore on Friday October 06, 2023 @02:56PM (#37794730)
All the more reason to rewrite it... You don't want some mistake in 30 year old COBOL code to be making your 2023 experiment to have wrong results.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by mlyle on Friday October 06, 2023 @03:25PM (#37795088)
The whole point is in these older numerical codes is that they're proven and there's a long history of results to compare against.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by gmueckl on Friday October 06, 2023 @03:12PM (#37794930)
That's the complete opposite of what is actually the case: some of that really old code in these programs is battle-tested and verified. Any rewrite of such parts would just destroy that work for no good reason.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by dpe82 on Friday October 06, 2023 @03:54PM (#37795456)
*FORTRAN.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by _a_a_a_ on Friday October 06, 2023 @04:20PM (#37795825)
Why don't YOU take some old code and rewrite it. I tried it for some 30+ year old HPC code and it was a grim experience and I failed hard. So why not keep your lazy, fatuous suggestions to yourself.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by throwaway10965 on Friday October 06, 2023 @02:55PM (#37794719)
Sounds like a great job for LLMs. Are there any public repositories of this code? I want to try.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by mlyle on Friday October 06, 2023 @05:50PM (#37796887)
Sounds like a -terrible- job for LLMs, because this is all about attention to detail. Order of operations and specific constructs of how floating point work in the codes in question are usually critical.

Have fun: https://www.qsl.net/m5aiq/nec-code/nec2-1.2.1.2.f

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by throwaway10965 on Saturday October 07, 2023 @06:13AM (#37800487)
Attention to detail can come later when there's something that humans can get started with. I did not mean that LLM could do it all alone.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by mlyle on Saturday October 07, 2023 @01:20PM (#37803577)
A human has to have the knowledge of what the code is trying to do and what the requisites are for accuracy and numerical stability. There's no substitute for that. Having a translation aid doesn't help at all unless it's perfect: it's more work to verify the output from a flawed tool than to do it right in this case.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by hulitu on Saturday October 07, 2023 @04:56PM (#37805485)
CUDA ? I thought rust was the future. /s
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by xadhominemx on Friday October 06, 2023 @12:00PM (#37792492)
Agreed, the CPUs are not performing the scientific calculations in this system.

Also note — this project is quite modest in scale. Dozens of GenAI clusters larger than this computer will be installed at cloud data centers in the next 18 months.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by hexane360 on Friday October 06, 2023 @12:38PM (#37792988)
[delayed]
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by RetroTechie on Friday October 06, 2023 @01:07PM (#37793321)
"The Jupiter will instead have SiPearl’s ARM processor based on ARM’s Neoverse V1 CPU design. SiPearl has designed the Rhea chip to be universally compliant with many accelerators, and it supports high-bandwidth memory and DDR5 memory channels."

And

"Jülich is also building out its machine-learning and quantum computing infrastructure, which the supercomputing center hopes to plug in as accelerator modules hosted at its facility."

So a modular setup, where different aspects can be upgraded as needed. Btw:

> Also note — this project is quite modest in scale.

"Exascale" and €273M doesn't sound modest to me. No matter what it's compared against.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by pclmulqdq on Friday October 06, 2023 @01:36PM (#37793642)
Almost all of the AI computers being built now are relatively modestly sized compared to a supercomputer. All but the biggest ones are at or under the low hundreds of nodes (low thousands of GPUs). The only real exceptions are the few AI hyperscale companies that want to sell GPU computing to others.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by bee_rider on Friday October 06, 2023 @02:17PM (#37794206)
Do AI hyperscalers devote a their whole system to one big run anyway?

If they don’t, then those are big clusters in the sense that AWS is the world’s biggest supercomputer, which is to say, not.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by pclmulqdq on Friday October 06, 2023 @05:11PM (#37796473)
AWS is not a supercomputer because it doesn't have high-adjacency networking. If AWS turned its biggest region loose on Linpack, I would be surprised if they cracked the top 50 on the supercomputer list, despite probably having more cores than #1.

The AI hyperscalers certainly claim to be able to devote 100% of cluster capacity to one training run. Google is training some huge models, OpenAI is also.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by xadhominemx on Friday October 06, 2023 @02:27PM (#37794346)
At €300m, the Jupiter will be ~10,000 H100s. Each major CSP will have several clusters this size or larger within a few quarters.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by bee_rider on Friday October 06, 2023 @01:11PM (#37793368)
Will these GenAI clusters have similar interconnects and ability to run scientific computing/HPC codes? AI has moved over to ASICS and GPUs nowadays, right? I also have no idea what their interconnect requirements are, but the task seems pretty low communication, I wonder if they can get by with a cheaper interconnect.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by the_svd_doctor on Friday October 06, 2023 @02:34PM (#37794450)
AI training requires a lot of global reductions which must be very fast otherwise everything slows down. So they also require fast and low latency interconnects.
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by brucethemoose2 on Friday October 06, 2023 @12:00PM (#37792493)
SIMD heavy CPUs can provide quite respectable HPC throughout.

The US Dept of Energy had very favorable things to say about the Fujitsu A64FX, which is architecturally similar to the SiPearl Rhea (HBM memory, ARM SVE happy, fast interconnect): https://www.osti.gov/biblio/1965278

They seemed to like the easy porting and flexible programming (since its "just" CPU SIMD) and specifically describe it as competitive with Nvidia:

> To highlight, the pink line represents the energy efficiency metric for A64FX in boost power mode (described in Section IV-C) with an estimated TDP of 140 W and surpassed by the red and yellow lines that represent data for the Volta V100 GPU (highest) and KNL, respectively. The A64FX architecture scores better with the energy efficiency metric relative to the performance efficiency metric due to its low power consumption.

In fact, ARM A64FX supercomputers topped the Green500 for some time, which is the global supercomputer power efficiency ranking, outclassing Nvidia/Intel/AMD machines.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by monocasa on Friday October 06, 2023 @12:27PM (#37792852)
You'd be surprised. A lot of supercomputers aren't that much about individual CPU core perf, but having a lot of low power cores connected in a novel way. The BlueGene supercomputers were composed of low spec PowerPC cores (even for the time).

High perf/watt matter more than just high perf/node, but even that balanced against 'how low latency can the interconnect be'.

You then hit the high FLOP count with tons of nodes.

To be fair Nvidia realized this paradigm years ago too, which is why they bought Mellanox.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by fsckboy on Sunday October 08, 2023 @03:56PM (#37813867)
I can believe what you are saying, but what are you saying, is it heat (ie cooling), power (cost), or hw cost/flop (because these chips are cheaper than screamers) that makes this an optimal solution?
Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by monocasa on Monday October 09, 2023 @04:29PM (#37825102)
I'd describe the end goal as TCO of achievable FLOP.

So a lot of factors come out of that, and a lot of designs that take interesting stabs at new balances towards that goal.

Re:EU Grabs ARM for First ExaFLOP Supercomputer, x86 Misses Out by cmdrk on Friday October 06, 2023 @12:43PM (#37793047)
it really depends on the workload. as other posters have said, not everything can/should be ported to GPU. some scientific calculations are simply not parallelizable in that way.

typically at least in the US there's a mix of GPU-focused machines as well as traditional CPU-focused machines. the leadership class machines (i.e., the machines funded to push the FLOPS records) tend to be highly focused on GPU. one reason is fixed cooling/power availability. I assume these facilities are looking at ARM as a way to save 10-20% on power and thus cram that much more into the facility.