In a shared-memory multiprocessor, all cores and all memory are logically connected along a massive shared bus, and both inter-core communication and regular core-to-main-memory communication takes up slots on the bus. In practice, of course, it isn't physically a single bus but a more complex topology (usually meshes, I think) with protocols that make it like a bus for cache coherency purposes.

One of the readily apparent consequences of interconnect is NUMA, non-uniform memory accesses. Historically (my information on this matter is a few years out of date, so I don't know how much this is true today), this has been a bigger issue for AMD than Intel, as AMD had wildly divergent memory latencies (2-3× between worst/best case) even for a 4-core processor. So if the OS decided to resume execution of a sleeping program on a different core, your performance might suddenly drop by ½ or ⅓.

For some benchmarks (such as classic SPEC, or make -j), multicore performance is approximated by running the same binary on every core and there is no sharing between different copies. The interconnect here matters only insomuch as the cross-chatter on the interconnect may impede performance. But HPC applications tend to be much more sensitive to the actual interconnect, since reading and writing from other processors' memory is more common (transferring halo data, or having global data structures that you need to read from and occasionally write).

The basic theory is called "MESI": Modified / Exclusive / Shared / Invalid. When a remote core reads a cache-line, if no one else has that cache-line then its the Exclusive owner. But if multiple caches have the cache-line, then it is a Shared cache line instead. If a remote cache changes, then your current cache-line becomes Invalid, while the cache line that's valid is placed into the Modified state (cache is correct, RAM is incorrect).

MESI isn't how things are done in practice: additional messages and states are piled on top in proprietary ways. But MESI will get you to the "basic textbook" concept at least.

EDIT: Oh, and "snooping". Caches "snoop" on each other's communications, which helps keep the MESI state up to date. Now you can have directory-based snooping (kind of a centralized push), or rings, or whatever. Those details are proprietary and change from chip to chip. In practice, these details are almost never published, so your best bet to understanding is "Something MESI-like is going on..."

It's (relatively) easy to have lots of cores on one die, it's not so easy to keep them fed.

Early AMD Softiron and Applied Micro boards (which had 8 cores unlike the ThunderX which had 48 or 96) were actually faster, which I always found interesting.

But Ampere's previous generation (before N1) is fast, much faster than the ThunderX2. Afaik, they built it on top of previous Applied Micro IP. So I'd expect N1 to be in a different league and not worth comparing to ThunderX2.

I think you're getting ThunderX2 and ThunderX mixed up here.

I wouldn't say it's the same order of magnitude for the 2 comparisons, but it's definitely noticeable.

The connection fabric is utterly important, and expensive.

May have consumed far less power doing the job though. For server farms performance-per-watt can be far more important than absolute speed per unit, especially for massively parallel tasks.

No one answered that and from the core to core ping-pong test the second processor their is some weird, possibly performance crushing issues.