The VM system could quite simply track the physical addresses as a pair of `u64_t`s or whatever, and present those pages as 64-bit pointers.
It seems in particular you might want to have this anyways, because the actual costs for dealing with such external memories would have to be much higher than local memory. Optimizing access would likely involve complicated cache hierarchies.
I mean, it'd be exciting if we had need for memory space larger than 2^64 but I just find it implausible with current physics and programs? But I'm also getting old.
> If everyone has the same address space, then you can share pointers / graphs between nodes and the underlying routing/ethernet software will be passing the data automatically between all systems. Its actually quite convenient.
Having more space available usually opens up more interesting possibilities. I'm going to rattle off some assorted options. If there's multiple paths to a given bit of data, we could use different addresses to refer to different paths. We could do something like ILA in IPv6, using some of the address as a location identifier: having enough bits for both the location and the identity parts of the address without being too constrained would be helpful. We could use the extra pointer bits for tagged memory or something like CHERI, which allow all kinds of access-control or permission or security capabilities. Perhaps we create something like id's MegaTexture, where we can procedurally generate data on the fly if given an address. There's five options for why you'd want more address space than addressable storage. And I think some folks are already going to be quite limited & have quite a lot of difficulty partitioning up their address space, if they only have for example 1.6m buckets of 1TB (one possible partitioning scheme).
The idea of being able to refer to everything anywhere that does or did exist across a very large space sure seems compelling & interesting to me!
It's a disaster of latency issues too, but it's not like that's surprising anyone either, and we already have NUMA on some multi-core systems which is the same problem.
We have existing tools that can be extended in straightforward ways to deal with these issues. And it's not like there's a silver bullet here; having separate address spaces everywhere comes with its own disaster of issues. Pick your poison.
I'm not advocating 128-bit pointers, or saying they're useful or realistic. I'm just saying, what if.
(For that matter, what happens when different segments of memory have complex access controls? What about needing to retry to failures like network partitions that don’t happen in a normal memory space?)
If latency matters, which it usually does, then you need some kind of memory access hierarchy, copying things back and forth, etc. and then you’ll almost certainly need a library of some kind to manage all this, prefetch from a slow range of memory and populate some of your fast local memory, etc.
At that point, we’ve done a lot of work, and are still pretending there is no network or the like, even though it’s there. It isn’t free, anyway. We’d also need checksums, cross network/fabric/access error handling, etc.
And with 128 bits, we could also use something like IPv6 with the lower 64 bits being byte address hah.
The HPC world is all about RDMA to direct-access their huge data sets, and likely hyperscaler clouds are starting to do that too.
The latency gap is just treated as yet another layer of the cache model.
And we already have error correction for local RAM.
Those were due to bottlenecks that don’t exist anymore, and even in your example were even then only emergency measures due to local resource shortages.
ECC is also not reliable/sufficient in the face of issues that arrive when networks start playing their part.
Having a single system image has many potential upsides, and understandability & reasonability are high among them.
At that point, maybe you want 256bit or 512bit pointers so that you can build a single global addressable system for all memory in the world.
In the Grace Hopper whitepaper, NVIDIA says that they connect multiple nodes with a fabric that allows them to creat a virtual address space across all of them.
Again, I'm not a supercomputer programmer. But the whitepapers often discuss RDMA.
From my imagination, it sounds like any other "mmap". You, the programmer, just remembers that the mmap'd region is slower (since it is read/write to a Disk, rather than to RAM). Otherwise, you treat it "like RAM" from a programming perspective entirely for convenience sake.
As long as you know my_mmap_region->next = foobar(); is a slow I/O operation pretending to be memory, you're fine.
---------
Modern systems are converging upon this "single address space" programming model. PCIe 3.0 implements atomic operations and memory barriers, CXL is going to add cache-coherence over a remote / I/O interface. This means that all your memory_barriers / atomics / synchronization can be atomic-operations, and the OS will automatically translate these memory commands into the proper I/O level atomics/barriers to ensure proper synchronization.
This is all very new, only within the past few years. But I think its one of the most exciting things about modern computer design.
Yes, its slow. But its consistent and accurately modeled by all elements in the chain. Atomic-compare-and-swap over RDMA can allow for cache-coherent communications and synchronization over Ethernet, over GPUs, over CPUs, and any other accelerators sharing the same 64-bit memory space. Maybe not quite today, but soon.
This technology already exists for PCIe 3.0 CPU+GPUs synchronization from 8 years ago (Shared Virtual Memory). Its exciting to see it extend out into more I/O devices.
Caring about where in memory your data lives is different from dealing with cache or paging. Programmers have to plan ahead to keep frequently accessed data in fast RAM, and infrequently accessed data in "slow" RAM. You'll probably need special APIs to allocate and manage memory in the different pools, not unlike the Address Windowing Extensions API in Microsoft Windows.
And once you extend "memory" outside the chassis, you'll have to design your application with the expectation that any memory access could fail because a network failure means the memory is no longer accessible.
If you only plan to deploy in a data center then maybe you can ignore pointer faults, but that is still a risk, especially if you decide to deploy something like Chaos Monkey to test your fault tolerance.
You have to deal with these things anyway in any kind of distributed setting. What this kind of location-independence via SSI really buys you is the ability to scale the exact same workloads down to a single cluster or even a single node when feasible, while keeping an efficient shared-memory programming model instead of doing slow explicit message passing. It seems like a pretty big simplification.
The marketing for SSI was that it was simple because programmers would not have to learn explicit message passing. Unfortunately, people that buy supercomputers tend to care about performance, so designing a supercomputer that is difficult to optimize misses the point. In real code, the only way to make them perform well was to layer topology-aware message passing on top of the shared memory model. At which point you should've just bought a message passing architecture.
There is only one type of large-scale SSI architecture that is able to somewhat maintain the illusion of uniform shared memory -- hardware latency-hiding e.g. barrel processors. If programmers have difficulty writing scalable code with message passing, then they definitely are going to struggle with this. These systems use a completely foreign programming paradigm that looks deceptively like vanilla C++. Exceptional efficient, and companies design new ones every few years, but without programmers that grok how to write optimal code they aren't much use.
It might make sense for large distributed systems to move to a 128-bit architecture, but I don't see any reason for consumer devices, at least with current technology.
No its not. That's why Kogge-Stone's carry lookahead adder was such an amazing result. O(log(n)) latency with respect to operand width with O(n) total half-adders used.
It may seem like its super-linear. But the power of prefix-sums leads to a spectacular and elegant solution. Kogge-stone (and the concept of prefix-sums) is one of the most important parallel-programming / parallel-system results of the last 50 years. Dare I say it, its _THE_ most important parallel programming concept.
> multiplication latency
You could just... not implement 128-bit multiplication. Just support 128-bit pointers (aka: addition) and leave multiplication for 64-bits and below.
I know it was one of the first time I thought to myself: this is not just a straightforward pipeline, yet it all follows such a beautifully geometrical interconnect pattern. Super fast, yet very elegant to layout.
"A Parallel Algorithm for the Efficient Solution of a General Class of Recurrence Equation", by Kogge and Stone.
It proves the result for _all_ associative operations (technically, a class slightly larger than associative. Kogge and Stone called this a "semi-associative" operation).
A bit sad that 1974 papers are still behind a IEEE paywall...
Edit: Just finished reading it. I have to say that the generalization of 3.2 got a bit over me, but otherwise it's pretty amazing that they could define such a generalization. Intuition for those type of problem is often to proceed one step at a time, N times.
That it is provably doable in log2(N) is great, especially since it allows for a choice of the depth/number of processors you want to use for the problem. Hopefully next time I design a latency-constrained system I remember to look at that article
Nah. Your next step is to read "Data parallel algorithms" by Hillis and Steele, which starts to show how these principles can be applied to code. (Much higher-level, easier to follow, paper. From ACM too, so its free since its older than 2000)
Then you realize that all you're doing is following the steps towards "Map-reduce" and modern parallel code and just use Map Reduce / NVidia cub::scan / etc. etc. and all the modern stuff that is built from these fundamental concepts.
Kogge and Stone's paper sits at the root of it all though.
I feel like this sums up the way of thinking one must have in this paradigm.
It's funny because when designing digital hardware, you're kind-of trained to see things under that angle, since often the commands you write will expand in that tree-like structure, but to gates/ALU instead of data.
Then managing the data-path often piggy-backs on the hardware structure you just generated.
I feel like I have a overall grasp on these concept, yet there's so much interesting results that I don't know about...
There's going to be no issues implementing a 128-bit adder. None at all.
Transistors have a "latency" measured in picoseconds, not nanoseconds.
A 128-bit adder will have a latency of 1-clock cycle, a 10% increase from a 64-bit adder of latency of 1-clock cycle. Aka: it doesn't matter cause it'd almost certainly fit inside of the clock cycle of modern CPUs.
And seeing datacenter after datacenter shooting up like mushrooms, there might be some sort of abstraction running in this direction, that makes 128bit addresses feasible. At the moment 64bit seems like paging in this sense.
I nominally agree with most of your post. But I should note that modern systems seem to be moving towards a "one pointer space" for the entire cluster. For example, 8 GPUs + 2 CPUs would share the same virtual memory space (GPU#1 may take one slice, GPU#2 takes another, etc. etc.).
This allows for RDMA (ie: mmap across Ethernet and other networking technologies). If everyone has the same address space, then you can share pointers / graphs between nodes and the underlying routing/ethernet software will be passing the data automatically between all systems. Its actually quite convenient.
I don't know how the supercomputer software works, but I can imagine that 4000 CPUs + 16000 GPUs all sharing the same 64-bit address space.