First, you claimed that there was no L3 BW test. Now, I am not even sure if you're trolling me or lacking knowledge or what at this point?
Please do tell what you consider a "proper test of L3 cache"? And why do you consider their test invalid?
I am curious because triggering 32 physical core threads to run over 32 independent chunks of data (totaling 3G and not 128M) seems like a pretty valid read BW experiment to me.
> Sapphire Rapids uses Golden Cove cores.
Right, but you missed the part that former is configured for the server market and the latter for the client market. Two different things, two different chips, different memory controllers if you wish. That's why you cannot compare one to each other directly without caveats.
Chips and Cheese are actually guilty of doing that but it's because they're lacking more HW to compare against. So some figures that you find in their articles can be misleading if you are not aware of it.
> You seem to view CPU performance as being determined by memory bandwidth rather than computational ability.
But that's what you said trying to refute the fact why Intel was in a lead over AMD up until zen5? You're claiming that AVX-512 workloads and load-store BW are largely irrelevant because CPUs are anyway bottlenecked by the system memory bandwidth.
> That would be true if the primary workload of CPUs were memory bandwidth bound workloads, but it is not since the primary workloads of CPUs is compute bound workloads. Thus, how fast CPUs read from memory does not really matter for CPU workloads.
I am all ears to hear what datacenter workloads you have in mind that are CPU-bound?
Any workload besides the most simplest one is at some point bound by the memory BW.
> The purpose of a CPU’s cache is to reduce the von Neumann bottleneck by cutting memory access latency.
> That way the CPU core spends less time waiting before it can use the data and it can move on to a subsequent calculation.
> How much memory throughput CPUs get from L1 cache is irrelevant to CPU performance outside of exceptional circumstances.
You're contradicting your own claims by saying that cache is there to hide (cut) the latency but then you continue to say that this is irrelevant. Not sure what else to say here.
> but they are truly exceptional since any importan workload where memory bandwidth matters is offloaded to a GPU because a GPU often has 1 to 2 orders of magnitude more memory bandwidth than a CPU.
99% of the datacenter machines are not attached to the GPU. Does that mean that 99% of datacenter workloads are not "truly exceptional" for whatever the definition of that formulation and they are therefore mostly CPU bound?
Or do you think they might be memory-bound but are missing out for not being offloaded to the GPU?
I claimed that they did not provide figures for L3 cache bandwidth. They did not.
> Now, I am not even sure if you're trolling me or lacking knowledge or what at this point?
You really should be grateful that a professional is taking time out of his day to explain things that you clearly do not understand.
> Please do tell what you consider a "proper test of L3 cache"? And why do you consider their test invalid?
You cannot measure L3 cache performance by measuring the bandwidth on a region of memory larger than the L3 cache. What they did is a partially cached test and it does not necessarily reflect the true L3 cache performance.
> I am curious because triggering 32 physical core threads to run over 32 independent chunks of data (totaling 3G and not 128M) seems like a pretty valid read BW experiment to me.
This does not test L3 cache bandwidth at all. This would be a generic memory bandwidth test. Their graphs showing performance at different amounts of memory is designed to show the performance of the memory hierarchy. The moment you exceed the amount of cache at a certain level, the performance transitions to different level. They did benchmarks on different amounts of memory to get the points in their graph and connected them.
> Right, but you missed the part that former is configured for the server market and the latter for the client market. Two different things, two different chips, different memory controllers if you wish. That's why you cannot compare one to each other directly without caveats.
The Xeon Max chips with its HBM2e memory is the one place where 2 AVX-512 loads per cycle could be expected to be useful, but due to internal bottlenecks they are not.
Also, for what it is worth, Intel treats AVX-512 as a server only feature these days, so if you are talking about Intel CPUs and AVX-512, you are talking about servers.
> But that's what you said trying to refute the fact why Intel was in a lead over AMD up until zen5? You're claiming that AVX-512 workloads and load-store BW are largely irrelevant because CPUs are anyway bottlenecked by the system memory bandwidth.
I never claimed AVX-512 workloads were irrelevant. I claimed doing more than 1 load per cycle on AVX-512 was irrelevant because you are going to be bottlenecked on the first load operation for anything that is memory bandwidth bound. Intel losing its lead to AMD is due to entirely different reasons than AVX-512 loads per cycle. This is obvious when you consider that most workloads do not touch AVX-512.
To be clear, when you can use AVX-512, it is useful, but the ability to do 2 loads per cycle does not add to the usefulness very much.
> I am all ears to hear what datacenter workloads you have in mind that are CPU-bound?
Note that I said compute bound, not CPU bound. Nearly all datacenter workloads run on CPUs are compute bound. Memory bandwidth bound things are better run on GPUs, which have far more memory bandwidth.
> Any workload besides the most simplest one is at some point bound by the memory BW.
This is wrong. It does not matter how many experts you consult. Every single one will try to explain the von Neumann bottleneck to you upon hearing this.
> You're contradicting your own claims by saying that cache is there to hide (cut) the latency but then you continue to say that this is irrelevant. Not sure what else to say here.
There is no contradiction. The cache is there to hide latency. It is largely not there to provide bandwidth. It only does as a side effect of being lower latency than system memory. The TACC explanation of how queuing theory applies to CPUs makes it very obvious that memory bandwidth is inversely proportional to memory access times, which is why the cache has more memory bandwidth than system RAM. It is a side effect of the actual purpose, which is to reduce memory latency.
> 99% of the datacenter machines are not attached to the GPU. Does that mean that 99% of datacenter workloads are not "truly exceptional" for whatever the definition of that formulation and they are therefore mostly CPU bound?
99% is a made up number. Your question is nonsensical.
Not only you're ending up being very disrespectful but you're also pulling out the appeal to authority argument. Also something Brendan Gregg did on me here at HN meaning that no experience can substitute the amount of ego in some guys.
FWIW you can be let assured that you can't match my experience but that's not the argument I would ever pull off. I like to be proved wrong. This is a way I learn new things. BTW I designed my first CPU 15 years ago but during the career I learned to put my ego aside, discuss objectively, think critically and learn on my own reasoning mistakes from other people. Many of these points you are obviously lacking so this is a waste of time for me - I see no way to drive this discussion further but thanks anyway.
They do not. The chip has 105MB L3 cache and they tested on 128MB of memory. This exceeds the size of L3 cache and thus, it is not a proper test of L3 cache.
> If you look closely into my comment you're referring to you will see that I explicitly referred to Golden Cove core and not to the Sapphire Rapids core. I am not being pedantic here but they're actually different things.
Sapphire Rapids uses Golden Cove cores.
> And yes, Sapphire Rapids reach 450 GB/s in L1 for AVX-512 workloads. But SPR core is also clocked @3.8Ghz which is much much less than what the Golden Cove core is clocked at - @5.2GHz. And this is where the difference of ~200 GB/s comes from.
This would explain the discrepancy between your calculation and the L1 cache performance, although being able to get that level of bandwidth only out of L1 cache is not very useful for the reasons I stated.
> I'm not sure I follow this reasoning but if true then AMD and Intel have nothing to compete against each other since by the logic of CPU caches being limited in applicability, their designs are equally good for the most $$$ workloads.
Your reasoning was that the cores hypothetically being able to load/store AVX-512 faster was important for performance. The reality is that this does not really matter because of bottlenecks in the memory hierarchy. That is why Zen 4 did extremely well on AVX-512 code without the ability to do as many concurrent loads and stores.
Intel and AMD primarily compete in compute bound scenarios where memory bandwidth does not matter. This is why the vast amount of work they do on each CPU generation is designed to increase the amount of compute done per cycle on average (e.g. improved branch predictors, larger reorder buffers, wider cores, decreased memory access times from larger caches and better prefetch, etcetera). There do exist memory bound workloads (e.g. memset, memcpy, checksumming, GEMV, BLAS levels 1 and 2 in general...), but the memory bandwidth from L1 cache rarely matters for them since their working sets are often far too big. While there are still some memory bound operations done on CPUs, they are a small portion of what the CPUs do and the cases where they are not a small portion have long been moved to GPUs (e.g. inference, which primarily uses GEMV for token generation) since GPUs dwarf CPUs in memory bandwidth by 1-2 orders of magnitude.
L1 cache exists for latency and its high memory bandwidth is a side effect of that. Getting high memory bandwidth out of L1 cache does not matter as far as competitiveness goes since working sets are far bigger than the 80KB of L1 cache in Golden Cove. If working sets actually fit in L1 cache for its memory bandwidth to matter, there would be no need for the L2 cache and no need for the huge L3 cache (although perhaps a small cache shared by all cores could make sense). People would not bother building/buying machines with gigabytes or even terabytes of RAM either.