News for hackers. Stuff that matters.
Re:AMD 3rd Gen EPYC Milan Review by rbanffy on Monday March 15, 2021 @11:38AM (#26467253)
Why so little L3 cache on the competition?
Re:AMD 3rd Gen EPYC Milan Review by dragontamer on Monday March 15, 2021 @11:50AM (#26467419)
EPYC is a split L3 cache. Any particular core only benefits from 32MBs of L3, the 33rd MB is "on another chip". (EDIT: Zen2 was 16MBs, Zen3 is 32MBs. Fixed numbers for Zen3)

As such, AMD can make absolutely huge amounts of L3 cache (well, many parallel L3 clusters), while other CPU designers need to figure out how to combine the L3 so that a single core can benefit it from it all.

Re:AMD 3rd Gen EPYC Milan Review by rbanffy on Monday March 15, 2021 @01:24PM (#26468807)
What AMD does is not magic and is not beyond what others can do. My question is why they chose to have just 32MB for up to 80 cores when AMD can choose to have 32MB per 8-core chiplet.

As a comparison, an IBM z15 mainframe CPU has 10 cores and 256MB per socket.

Re:AMD 3rd Gen EPYC Milan Review by dragontamer on Monday March 15, 2021 @01:43PM (#26469034)
> As a comparison, an IBM z15 mainframe CPU has 10 cores and 256MB per socket.

Well, that's eDRAM magic, isn't it? Most manufacturers are unable to make eDRAM on a CPU.

> My question is why they chose to have just 32MB for up to 80 cores when AMD can choose to have 32MB per 8-core chiplet.

From my understanding, those ARM chips are largely I/O devices: read from disk -> output to Ethernet.

In contrast, IBM's are known for database backends, which likely benefits from gross amounts of L3 cache. EPYC is general purpose: you might run a database on it, you might run I/O constrained apps on it. So kind of a middle ground.

Re:AMD 3rd Gen EPYC Milan Review by floatboth on Monday March 15, 2021 @08:19PM (#26471272)
Didn't Intel push eDRAM magic into various laptop chips around Broadwell?

> those ARM chips are largely I/O devices

Anything Neoverse-N1 is pretty good at general compute and databases. There's definitely a lot of Postgres running on AWS Graviton2 instances already :)

Re:AMD 3rd Gen EPYC Milan Review by meepmorp on Monday March 15, 2021 @01:46PM (#26469071)
IBM doesn't fab its own chips, right? I thought they used GF.
Re:AMD 3rd Gen EPYC Milan Review by wmf on Monday March 15, 2021 @05:11PM (#26469143)
It's basically IBM fabs that were "sold" to GlobalFoundries. AFAIK IBM processors use a customized process that isn't used by any other GF customers.
Re:AMD 3rd Gen EPYC Milan Review by xirbeosbwo1234 on Monday March 15, 2021 @12:11PM (#26467760)
That's not quite accurate. Every core has access to the entire L3, including the L3 on an entirely different socket. CPUs communicate through caches, so if a core just plain couldn't talk to another core's cache then cache coherency algorithms wouldn't work. Though a core can access the entire cache, the latency is higher when going off-die. It is really high when going to another socket.

The first generation of Epyc had a complicated hierarchy that made latency quite hard to predict, but the new architecture is simpler. A CPU can talk to a cache in the same package but on a different die with reasonably low latency.

(I don't have numbers. Still reading.)

Re:AMD 3rd Gen EPYC Milan Review by dragontamer on Monday March 15, 2021 @12:22PM (#26467953)
In Zen1, the "remote L3" caches had longer read/write times than DDR4.

Think of the MESI messages that must happen before you can talk to a remote L3 cache:

1. Core#0 tries to talk to L3 cache associated with Core#17.

2. Core#17 has to evict data from L1 and L2, ensuring that its L3 cache is in fact up to date. During this time, Core#0 is stalled (or working on its hyperthread instead).

3. Once done, then Core#17's L3 cache can send the data to Core#0's L3 cache.

----------

In contrast, step#2 doesn't happen with raw DDR4 (no core owns the data).

This fact doesn't change with the new "star" architecture of Zen2 or Zen3. The I/O die just makes it a bit more efficient. I'd still expect remote L3 communications to be as slow, or slower, than DDR4.

Re:AMD 3rd Gen EPYC Milan Review by ChuckNorris89 on Monday March 15, 2021 @11:40AM (#26467278)
Limitations due to die size and power consumption since Intel Xeon is still on the ye olde 14nm++ process.

Also since Xeon dies are monolithic, unlike AMDs chiplet design, means that increasing the size of certain components on the die, like cache for example, increases the risk of defects wich reduces the yields, making them unprofitable.

Re:AMD 3rd Gen EPYC Milan Review by greggyb on Monday March 15, 2021 @09:18PM (#26471712)
FYI, 'ye olde' is actually using the letter 'y' to represent an Old English 'thorn' character. The thorn character represents the 'th' sound. Printing presses gained prominence in non-English speaking regions earlier than in English speaking regions. The 'y' was substituted for the 'thorn' due to some amount of similarity in shape.

Thus, 'ye olde' would be pronounced 'the old'. 'Ye' is just a different form of 'the' from a time when 'y' represented 'th'.

So, 'the ye olde' would be read as 'the the olde'.

Re:AMD 3rd Gen EPYC Milan Review by rbanffy on Monday March 15, 2021 @11:57AM (#26467549)
True, but the ARM ones have just 32MB for up to 80 threads.

I wonder if we could get numbers for L3 misses and cycles spent waiting for main memory under realistic workloads.

Re:AMD 3rd Gen EPYC Milan Review by dragontamer on Monday March 15, 2021 @05:23PM (#26469257)
That information changes with every application. Literally every single program in the world has its own cache characteristics.

I suggest learning to read performance counters, so that you can get information like this yourself! L3 cache is a bit difficult for AMD processors (because many cores share the L3 cache), but L2 cache is pretty easy to work with and profile.

General memory-reads / memory latency is pretty easy to read with various performance counters. Given the amount of latency, you can sorta guess if its in L3 or in DDR4.

Re:AMD 3rd Gen EPYC Milan Review by xirbeosbwo1234 on Monday March 15, 2021 @11:59AM (#26467591)
First off, it's not a direct comparison. The Epyc has one L3 cache per chiplet. This means that latency is not uniform across the entire L3 cache. This was a serious concern on the first generation of Epyc, where accessing L3 could take anywhere from zero to three hops across an internal network. AMD has greatly improved the problem on the more recent generations by switching to a star topology with more predictable latency.

That said, there are two major reasons:

1. Epyc is on a chiplet architecture. Large chips are harder to make than small ones. Building two 200mm^2 chips is cheaper than building one 400mm^2 chip. Since Epyc has a chiplet architecture, this means they can put more silicon into a chip for the same price. This means that Epyc can be just plain bigger than the competition. This comes with some complexity and inefficiency but has, so far, paid off in spades for them.

2. Epyc is on a newer process. This means AMD can fit more transistors even with the same area. Intel has had serious problems with their newer processes, so this is not an advantage AMD expected to have when designing the part. The use of a cutting-edge process was, in part, enabled by the chiplet architecture. It is possible to fabricate several small chips on a 7nm process even though one large chip would be prohibitively expensive, and AMD has been able to use a 14nm process in parts of the CPU that wouldn't benefit from a 7nm process to cut costs.

The first point is serious cleverness on the part of AMD. The second point is mostly that Intel dropped the ball.

Re:AMD 3rd Gen EPYC Milan Review by totalZero on Monday March 15, 2021 @12:13PM (#26467794)
What is the likelihood that mixed-process chiplets become the state of the art?
Re:AMD 3rd Gen EPYC Milan Review by Macha on Monday March 15, 2021 @01:15PM (#26468680)
Aren't they already for big (desktop/workstation/server) chips? I'd say Zen3 is the state of the art in that market and that uses a mixed process. The IO dies are global foundries 12nm for AMD.

The mobile market cares more about efficiency than easily scaling up to much bigger chips, so the M1 and other ARM chips are probably going to ignore this without much consequence for smaller chips.

Intel still tops sales because of non-perf related reasons like refresh cycles, distrust of AMD from last time they fell apart in the server space, producing chips in sufficient quantities unlike the entire rest of the industry fighting over TSMC's capacity, etc.

Re:AMD 3rd Gen EPYC Milan Review by uluyol on Monday March 15, 2021 @01:11PM (#26468611)
Intel already said they would use chiplets [1] and TSMC has been talking about the various packaging technologies being developed [2].

[1] https://www.anandtech.com/show/16021/intel-moving-to-chiplet...

[2] https://www.anandtech.com/show/16051/3dfabric-the-home-for-t...