News for hackers. Stuff that matters.
Re:Graviton2 and Graviton3 by erulabs on Saturday December 04, 2021 @06:32PM (#29445303)
> most workloads could actually run more efficiently if they had more memory bandwidth and lower latency access to memory

Turns out memory access speed is more or less the entire game for everything except scientific computing or insanely optimized code. In the real world, CPU frequency seems to matter much less than DRAM timings, for example, in everything but extremely well engineered games. It'll be interesting to learn (if we ever do) how much of the "real-world" 25% performance gain is solely due to DDR5.

I remember getting my AMD K8 Opteron around 2003 or 2004 with the first on-die memory controller. Absolutely demolished Intel chips at the time in non-synthetic benchmarks.

Re:Graviton2 and Graviton3 by shoo on Saturday December 04, 2021 @06:44PM (#29445379)
> everything except scientific computing or insanely optimized code

for insanely unoptimized code, such as accidentally ending up writing something compute intensive in pure python, its very plausible for it to be compute constrained -- but less because of the hardware and more because 99 %or 99.9% of the operations you're asking the cpu to perform are effectively waste.

Re:Graviton2 and Graviton3 by throwaway894345 on Saturday December 04, 2021 @06:49PM (#29445408)
I'm skeptical of this. I would expect many of those wasted instructions to be expensive memory instructions. I'm pretty sure regular Python code allocates a lot and the interpreter spends lots of time chasing after memory references.
Re:Graviton2 and Graviton3 by erulabs on Saturday December 04, 2021 @06:47PM (#29445398)
Certainly, but many "wasteful" operations are reading/writing main memory unnecessarily - and since a single trip out to memory and back can take many many CPU cycles, typically optimizing memory access time does more for "bad code" than optimizing the number of CPU cycles per second. But obviously, you're right too - faster is faster after all :)
Re:Graviton2 and Graviton3 by klelatti on Saturday December 04, 2021 @06:51PM (#29445414)
The team that designed the original Arm CPU in 1985 came to the conclusion that bandwidth was the most important factor influencing performance - they even approached Intel for a 286 with more memory bandwidth!
Re:Graviton2 and Graviton3 by hinkley on Saturday December 04, 2021 @06:48PM (#29445403)
In the 90’s there were people trying to solve this problem by putting a small CPU on chip with the memory and running some operations there. I routinely wonder why memory hasn’t gotten smarter over time.
Re:Graviton2 and Graviton3 by eropple on Saturday December 04, 2021 @06:54PM (#29445437)
Distributing work to leverage faster memory locality is hard. It's not quite what you're talking about, but consider the Cell processor used in the PS3 - the compute capability in HTPC space was supposed to be prodigious, but even having faster (streaming) access to RAM had the tradeoff of dealing with code dispatch. (It's not a perfect example, the SPE model was also just kind of a pain, but you have to think about how to get your code local to the memory you want, how to keep allocation nonrivalrous, etc. - it's a lot!)
Re:Graviton2 and Graviton3 by erulabs on Saturday December 04, 2021 @06:59PM (#29445464)
Heh, funny you mention this, because I was _so excited_ about the CELL architecture when it was announced - exactly because of memory read/write speed. Then I tried to actually write some code for it, got depressed, and moved back to x86 for the next decade :(
Re:Graviton2 and Graviton3 by hinkley on Sunday December 05, 2021 @05:29PM (#29453983)
I suspect some future improvement on borrow checkers will facilitate doing this to a degree. But it's likely to be one of those things that only comes into being when someone needs it very badly.
Re:Graviton2 and Graviton3 by y4mi on Sunday December 05, 2021 @12:21AM (#29446946)
DDR5 has basically minor cpus on the RAM for improved performance. That's the main reason why they're so expensive and sometimes need active cooling
Re:Graviton2 and Graviton3 by Terry_Roll on Sunday December 05, 2021 @04:24AM (#29447938)
So like a (micro)SD card that has some arm cpu for wear levelling and other performance enhancements.

It sounds interesting. https://www.bunniestudios.com/blog/?p=3554

Re:Graviton2 and Graviton3 by GordonS on Sunday December 05, 2021 @04:54AM (#29448059)
Interesting - do you know what kind of performance gains DDR5 will provide for real-life workloads?

Having active cooling, making the system noisier and potentially hotter, seems like a pretty big down side.

Re:Graviton2 and Graviton3 by vlangber on Sunday December 05, 2021 @08:40AM (#29449178)
Anandtech compared DDR4 and DDR5 performance in their Alder Lake review:

https://www.anandtech.com/show/17047/the-intel-12th-gen-core...

Re:Graviton2 and Graviton3 by spenczar5 on Sunday December 05, 2021 @01:12AM (#29447177)
Man that sounds so hard to program. I wonder if that's the reason.
Re:Graviton2 and Graviton3 by baybal2 on Saturday December 04, 2021 @07:25PM (#29445624)
Memory bandwidth is not a problem. Even puny 1 channel memory desktops don't usually saturate it.

It's memory latency.

Re:Graviton2 and Graviton3 by tromp on Sunday December 05, 2021 @03:52AM (#29447814)
For some problems, there is a choice of how to organize its data structures. One that requires random access, and another that is mostly sequential access.

The latter might take an order of magnitude more space, while still being faster.

An example of such a problem is the Cuckoo Cycle Proof-of-Work [1].

[1] https://github.com/tromp/cuckoo

Re:Graviton2 and Graviton3 by mrcode007 on Saturday December 04, 2021 @09:08PM (#29446145)
Certain instruction sequences prohibit you from fully utilizing memory bandwidth and decrease it by as much as 40-60% so this statement is not true. For example, not using store and load buffers in ways they were designed to be used will lead to subpar performance for no apparent reason
Re:Graviton2 and Graviton3 by ahartmetz on Saturday December 04, 2021 @09:45PM (#29446308)
Can you elaborate? I have a side project(1) where all profilers I've used give a very muddled picture, so I'm very interested in the question of what slows down code on a modern "big" CPU with wide dispatch, a few kB of decoded ops buffer and a lot of OoO hardware.

(1) It encodes and decodes a protocol from a potentially untrusted source, so there is obviously a lot of waiting for previous results. That much is clear, however I expected profilers to show me some causal link between serial nature and slow execution, but they don't. I have tried perf, Valgrind-Callgrind and AMD μProf (because I have a Ryzen CPU on both of my main private computers). I'm not sure if the tools suck, my test cases suck, or I just don't know how to interpret the tools' results - assignment of cost to lines of code seeming highly unreliable is my main problem. Maybe the stupid things (most of optimization is about not doing stupid things, after that it gets properly hard) that these profilers are designed to catch aren't the stupid or unavoidable things my code is doing.

Re:Graviton2 and Graviton3 by namibj on Saturday December 04, 2021 @10:50PM (#29446548)
KDAB's hotspot is quite nice for analyzing perf recordings, and I suggest looking at stall cycle and "cycles with less than X uops dispatched" events to sample on. Yes, attributing to lines in code is hard for optimized compiler output, but it can (in the continuous release/`hotspot-git` AUR package) attribute to the disassembly of a function.
Re:Graviton2 and Graviton3 by mrcode007 on Wednesday December 08, 2021 @02:38AM (#29482177)
One way of approaching the problem is with the top down cycle accounting methodology

You can find a nice spreadsheet from Intel here:

https://download.01.org/perfmon/TMA_Metrics.xlsx

Also using Intel tools gives you much clearer answers. There are a bunch of scripts on top of perf that try to automate this. You can find them in pmu-tools git repo.

The basic workflow is always the same. Find out which part of the pipeline gets bottlenecked. Is it stuck in PCIE IO, memory, instruction decoding, etc. in my experience most of the time it is memory due to heavy pointer dereferencing and it is just a sad state of programming languages these days.

There are a bunch of memory benchmark tests that demonstrate these effects very well in the NUMA tools package source repository.

The problem with memory being the slow part is that it affects instruction fetch/decode cycle as well.

Poor instruction selection and sequencing can impact throughout because of port saturation (e.g. in case of AVX2) and leave other dispatch ports idle, premature store buffer flushes where you send 1-entry updates instead of sending stuff in chunks, leaving 70% of store buffer bandwidth idle, etc.

There are a lot of foot guns in a modern processor and usually it’s a mixture of these problems with one being dominant. A lot of times it is not possible to completely address the problem without rebuilding the software from scratch and properly leveraging hardware knowledge from the beginning when architecting your code.

Re:Graviton2 and Graviton3 by adgjlsfhk1 on Saturday December 04, 2021 @08:50PM (#29446067)
For well optimized code it is.
Re:Graviton2 and Graviton3 by mda on Saturday December 04, 2021 @07:02PM (#29445476)
I find this claim hard to believe honestly,could you point to examples where performance is limited by Dram speed and not by cpu / caches? They must be applications with extremely bad design causing super low cache hits.
Re:Graviton2 and Graviton3 by erulabs on Saturday December 04, 2021 @07:06PM (#29445505)
> They must be applications with extremely bad design causing super low cache hits

Yep, this is exactly the case - also, on systems that are busy and context-switching often and thus flushing their cpu caches more frequently. Combine the two, busy systems running loads of un-optimized code, and boom, you have described how most computers run in the real world. This is why "synthetic" benchmarks, which are well designed code running on quiet machines more or less match up to CPU Frequency exclusively.

I don't really have any good charts to show you, but you might checkout an old review of the processor I mentioned as having one of the first on-die memory controllers: https://techreport.com/review/5655/amds-opteron-146-processo...

The AMD Opteron 240 1.4GHz keeps up with chips close to 2x it's frequency - and the memory access times are close to 1/2 as costly (ie: almost all the performance gain from 2x frequency is made up by 1/2 memory access time) - this makes sense, but remember these are well optimized applications (POV-Ray and Lightwave were extremely synthetic). In the real world, opening 10 misc windows applications from 2003, the K8 (particularly when overclocked) was a _beast_.

Re:Graviton2 and Graviton3 by mda on Saturday December 04, 2021 @11:20PM (#29446652)
Well, Opteron is an ancient processor, I don't think we can make any conclusion based on that. Today's server processors has enormous caches compared to Opteron.

Honestly in cases you mention, badly designed processes killing the Cpu, I fail to see how faster ram makes a huge difference.

Re:Graviton2 and Graviton3 by erulabs on Sunday December 05, 2021 @12:00AM (#29446822)
I mean - the people who designed Gravaton 3 seem to agree with my premise, so at least that’s some validation. Alternatively, do some CPU profiling with your workstation - a massive amount of time is simple waiting for memory returns.
Re:Graviton2 and Graviton3 by sroussey on Saturday December 04, 2021 @08:30PM (#29445982)
This is why you dedicate entire machines to the same kind of load. All application code or all database. There was a moment where people tried to integrate—which was indeed faster but only for very limited use cases.
Re:Graviton2 and Graviton3 by hexxagone on Saturday December 04, 2021 @09:44PM (#29446299)
In data compression, inverting a BWT with large blocks or using Context Mixing to compress large blocks (which requires huge context maps). These 2 cases require a lot of random memory accesses.
Re:Graviton2 and Graviton3 by therealcamino on Saturday December 04, 2021 @08:48PM (#29446061)
Anything where the working set is larger than L3 cache.
Re:Graviton2 and Graviton3 by staticassertion on Saturday December 04, 2021 @08:07PM (#29445857)
> They must be applications with extremely bad design causing super low cache hits.

So basically any program written in a language with pointer types exclusively.

Re:Graviton2 and Graviton3 by throwaway894345 on Saturday December 04, 2021 @09:47PM (#29446313)
I thought I’d heard that Java VMs go to great lengths to maintain cache coherence? I’d be curious to hear from the Lisp folks because I always hear that Lisps can be surprisingly performant.
Re:Graviton2 and Graviton3 by staticassertion on Sunday December 05, 2021 @01:51PM (#29451770)
There are some optimizations in the JVM that will improve cache coherence. Bump allocation helps, inlining and escape analysis help, etc.

In theory the GC can also rearrange memory to 'compact' it. I'm not aware of this optimization in practice.

Re:Graviton2 and Graviton3 by throwaway894345 on Sunday December 05, 2021 @02:30PM (#29452213)
As far as I know, all mainstream Java GCs are compacting; however, I don’t have an idea about the degree this improves cache coherence.
Re:Graviton2 and Graviton3 by mda on Saturday December 04, 2021 @11:22PM (#29446659)
Even then, today's cpus have enormous caches, and not all parts of program is pointers. you cant make a crappy application much faster just because you have faster ram.