PCIe is a pretty complicated protocol based on messages. Using this for RAM would mean the CPU and everything will be required to speak this protocol. Using PCIe for RAM will also have impact on latency, which is undesired.

Speed is not always the only thing.

> Speed is not always the only thing.

or perhaps more accurately, bandwidth is not the only goal.

Or perhaps more accurately, throughput is not the only goal.

Throughput, latency, cost. Pick 2.

I wonder if this is to help support a newish arch. like the one Microsoft and Intel are proposing (Taipei conf.) with each laptop/pc having a dedicated NPU, CPU, GPU to bring some of the Copilot button back on PC? Everything I see today I inadvertently read in "oh this will help me as a hobbyist". It generally doesn't. Probably just technology doing what it does and moving forward.

I've seen some very recent research proposing just that: https://arxiv.org/pdf/2305.05033.pdf . Of course still far from production, but interesting that in some cases it actually also improves latency.

It would be nicer to see the ability to extend GPUs with off-the-shelf RAM, think SO-DIMM clipped to the back of the GPU. Then being able to choose to buy 2 x 64 GB DIMMs or just a 8GB one, maybe also have some GBs of good RAM soldered onto the GPU like it is now. One can dream.

Why does this make me picture a bunch of GPU company execs in big leather chairs cackling around a conference table with a dead goat in the middle?

But then the vendors would not be able to differentiate (ie, markup) their products.

They currently charge thousands more for adding $75 of memory.

Wasn't this a thing in the past - like the 1990s? I know the Matrox G200 supported it, but there might've been others too.

I'm speculating here but I think the answer has more to do with latency than bandwidth.

Without being a domain expert, my intuition would be that PCIe is optimized for throughput over latency and there's probably a throughput compromise when you want low-latency access.

Yeah but DDR is starting to move to more complex modulations, CXL is bringing down PCIe latency, and PCIe is starting from a position of elevated competence when compared to other standards. For example, you might expect that PCIe obtains parallelism by sending different packets down different lanes but in fact it shreds packets across lanes specifically because of latency. When PCIe eats another standard, the average quality of the standards ecosystem generally goes up.

That said, memory latency is so important that even small sacrifices should be heavily scrutinized.

What's the latency comparison between the memory bus and PCIe?

I dug into this for a hn comment months ago, but I think it's 2-3 orders of magnitude difference in latency. RAM is measured in nanoseconds, PCIe is measured in microseconds

The difference is smaller than that if you optimize it. The memory bus is about 50 ns in latency, and PCIe you can get down to sub 500 ns.

I don't think you've checked those numbers. SSD access is in the order of 10-20 microseconds (10,000 - 20,000 ns) and memory bus access is ~10-15 nanoseconds.

Here's the comment I made a couple months ago when I looked up the numbers:

I keep hearing that, but that's simply not true. SSDs are fast, but they're several orders of magnitude slower than RAM, which is orders of magnitude slower than CPU Cache.

Samsung 990 Pro 2TB has a latency of 40 μs

DDR4-2133 with a CAS 15 has a latency of 14 nano seconds.

DDR4 latency is 0.035% of one of the fastest SSDs, or to put it another way, DDR4 is 2,857x faster than an SSD.

L1 cache is typically accessible in 4 clock cycles, in 4.8 ghz cpu like the i7-10700, L1 cache latency is sub 1ns.

I have absolutely checked those numbers, and I have written PCIe hardware cores and drivers before, as well as microbenchmarking CPUs pretty extensively.

I think you're mixing up a few things: CAS latency and total access latency of DRAM are not the same, and SSDs and generic PCIe devices are not the same. Most of SSD latency is in the SSD's firmware and accesses to the backing flash memory, not in the PCIe protocol itself - hence why the Intel Optane SSDs were super fast. Many NICs will advertise sub-microsecond round-trip time for example, and those are PCIe devices.

Most of DRAM access latency (and a decent chunk of access latency to low-latency PCIe devices) comes from the CPU's cache coherency network, queueing in the DRAM controllers, and opening of new rows. If you're thinking only of CAS latency, you are actually missing the vast majority of the latency involved in DRAM operations - it's the best-case scenario - you will only get the CAS latency if you are hitting an open row on an idle bus with a bank that is ready to accept an access.

CAS meaning Compare and Swap?

https://en.wikipedia.org/wiki/CAS_latency is how long it takes to read a word out of the active row. Async DRAM had "row access strobe" and "column access strobe" signals.

Synchronous DRAM (SDR and DDRx) still has a RAS wire (row address strobe) and CAS wire (column address strobe), but these are accessed synchronously - and also used to encode other commands.

DDR DRAM is organized into banks (for DDR4/5 these are grouped in bank groups), rows, and columns. A bank has a dedicated set of read/write circuits for its memory array, but rows and columns within a bank share a lot of circuitry. The read/write circuits are very slow and need to be charged up to perform an access, and each bank can only have one row open at a time. The I/O circuits are narrower than the memory array, which is what makes the columns out of the rows.

The best-case scenario is that a bank is precharged and the relevant row is open, so you just issue a read or write command (generally a "read/write with auto-precharge" to get ready for the next command) and the time from when that command is issued to when the data bus starts the transaction is the CAS latency. If you add in the burst size (which is 4 cycles for a double-data-rate burst length of 8) plus the signal propagation RTT to the memory, you get your best-case access latency.

The worst-case scenario for DDR is then that you have just issued a command to a bank and you need to read/write a different row on that bank. To do that, you need to wait out the bank precharge time and the row activation time, and then issue your read or write command. This adds a lot of waiting where the bank is effectively idle. Because of that wait time, memory controllers are very aggressive about reordering things to minimize the number of row activations, so you may find your access waiting in a queue for several other accesses, too.

Also, processors generally use a hash function to map logical addresses to physical DRAM banks, rows, and columns, but they set it up to optimize sequential access: address bits roughly map to (from least to most significant): memory channel -> bank -> column -> row. It is more complicated than that in reality, but that's not a bad way to think about it.

I will defer to your experience, seems you have more on depth knowledge on this than I do.

Wouldn't PCI as a "memory bus" just be an SSD connected to the PCI bus setup as a swap space, which you can already do?

No, NVMe SSDs are not memory mapped. While they max out at 14GB/s - giving you 56GB/s per PCIe 5.0 x16 slot - emulating memory/swapping is expensive.

Isn't that what CXL is?

CXL supports cache-coherent memories and even the numbers claimed in marketing materials are ~5x when it comes than DDR5 access times measured on desktop CPUs in microbenchmarks. It's still a lot better than having software handle the coherency protocol and could be really useful to allow either very large ccNUMA systems or memory capacity and bandwidth hungry accelerators to access the host DRAM as long they can tolerate the latency e.g. something along the lines of a several SX-Aurora cards hooked into a big host with a few TiB of DRAM spread over ≥16 DDR5 or later channels.

Latency

Even the fastest (as in lowest latency) CXL memories claim a ~200ns latency (~4x worse than DDR5) and it wouldn't be free.

200ns latency for communicating with another CPU or GPU device is pretty good. Just imagine how many microseconds it would take to do all of this over ethernet.

Or, we could make it a general purpose unified system bus... call it the Unibus, or something like that.

SerDes overhead and high pin count? PCIe is not a real "bus" architecture but parallelized hub architecture system. I don't know enough to explain why but these leads to higher power consumption than bus at scale. M.2 SSDs with 3.0 x4 links already had issues with controllers thermal throttling often explained as heat from on-chip serializer/deserializer.