News for hackers. Stuff that matters.
Re:The road to Zettalinux by jmillikin on Friday September 23, 2022 @09:30AM (#32951691)
The section about 128-bit pointers being necessary for expanded memory sizes is unconvincing -- 64 bits provides 16 EiB (16 x 1024 x 1024 x 1024 x 1 GiB), which is the sort of address space you might need for byte-level addressing of a warehouse full of high-density HDDs. Memory sizes don't grow like they used to, and it's difficult to imagine what kind of new physics would let someone fit that many bytes into a machine that's practical to control with a single Linux kernel instance.

CHERI is a much more interesting case, because it expands the definition of what a "pointer" is. Most low-level programmers think of pointers as just an address, but CHERI turns it into a sort of tuple of (address, bounds, permissions) -- every pointer is bounds-checked. The CHERI folks did some cleverness to pack that all into 128 bits, and I believe their demo platform uses 128-bit registers.

The article also touches on the UNIX-y assumption that `long` is pointer-sized. This is well known (and well hated) by anyone that has to port software from UNIX to Windows, where `long` and `int` are the same size, and `long long` is pointer-sized. I'm firmly in the camp of using fixed-size integers but the Linux kernel uses `long` all over the place, and unless they plan to do a mass migration to `intptr_t` it's difficult to imagine a solution that would let the same C code support 32-, 64-, and 128-bit platforms.

(comedy option: 32-bit int, 128-bit long, and 64-bit `unsigned middle`)

The article also mentions Rust types as helpful, but Rust has its own problems with big pointers because they inadvisably merged `size_t`, `ptrdiff_t`, and `intptr_t` into the same type. They're working on adding equivalent symbols to the FFI module[0], but untangling `usize` might not be possible at this point.

[0] https://github.com/rust-lang/rust/issues/88345

Re:The road to Zettalinux by dragontamer on Friday September 23, 2022 @09:41AM (#32951841)
> it's difficult to imagine what kind of new physics would let someone fit that many bytes into a machine that's practical to control with a single Linux kernel instance.

I nominally agree with most of your post. But I should note that modern systems seem to be moving towards a "one pointer space" for the entire cluster. For example, 8 GPUs + 2 CPUs would share the same virtual memory space (GPU#1 may take one slice, GPU#2 takes another, etc. etc.).

This allows for RDMA (ie: mmap across Ethernet and other networking technologies). If everyone has the same address space, then you can share pointers / graphs between nodes and the underlying routing/ethernet software will be passing the data automatically between all systems. Its actually quite convenient.

I don't know how the supercomputer software works, but I can imagine that 4000 CPUs + 16000 GPUs all sharing the same 64-bit address space.

Re:The road to Zettalinux by cmrdporcupine on Friday September 23, 2022 @09:50AM (#32951945)
It seems to me that such a memory space could be physically mapped quite large while still presenting 64-bit virtual memory addresses to the local node? How likely is it that any given node would be mapping out more than 2^64 bytes worth of virtual pages?

The VM system could quite simply track the physical addresses as a pair of `u64_t`s or whatever, and present those pages as 64-bit pointers.

It seems in particular you might want to have this anyways, because the actual costs for dealing with such external memories would have to be much higher than local memory. Optimizing access would likely involve complicated cache hierarchies.

I mean, it'd be exciting if we had need for memory space larger than 2^64 but I just find it implausible with current physics and programs? But I'm also getting old.

Re:The road to Zettalinux by rektide on Friday September 23, 2022 @10:05AM (#32952099)
Leaving cluster coherent address space behind - like you say - is doable. But you lose what the parent was saying:

> If everyone has the same address space, then you can share pointers / graphs between nodes and the underlying routing/ethernet software will be passing the data automatically between all systems. Its actually quite convenient.

Re:The road to Zettalinux by vlovich123 on Friday September 23, 2022 @01:24PM (#32954690)
Let's say you have nodes that have 10 TiB of RAM in them. You then need 1.6M nodes (not CPUs, but actual boxes) to use up 64bits of address space. It seems like the motivation is to continue to enable Top500 machines to scale. This wouldn't be coming to a commercial cloud offering for a long time.
Re:The road to Zettalinux by rektide on Friday September 23, 2022 @02:51PM (#32955656)
Why limit yourself to in-memory storage? I'd definitely assume we have all our storage content memory mapped onto our cluster too, in this world. People have been building exabyte (1M gigabytes) scale datacenters since well before 2010, and 16 exabytes, the current Linux limit according to the most upvoted post here, isn't that much more inconceivable.

Having more space available usually opens up more interesting possibilities. I'm going to rattle off some assorted options. If there's multiple paths to a given bit of data, we could use different addresses to refer to different paths. We could do something like ILA in IPv6, using some of the address as a location identifier: having enough bits for both the location and the identity parts of the address without being too constrained would be helpful. We could use the extra pointer bits for tagged memory or something like CHERI, which allow all kinds of access-control or permission or security capabilities. Perhaps we create something like id's MegaTexture, where we can procedurally generate data on the fly if given an address. There's five options for why you'd want more address space than addressable storage. And I think some folks are already going to be quite limited & have quite a lot of difficulty partitioning up their address space, if they only have for example 1.6m buckets of 1TB (one possible partitioning scheme).

The idea of being able to refer to everything anywhere that does or did exist across a very large space sure seems compelling & interesting to me!

Re:The road to Zettalinux by vlovich123 on Friday September 23, 2022 @06:44PM (#32957963)
Maybe. You are paying a significant performance penalty for ALL compute to provide that abstraction though.
Re:The road to Zettalinux by Aperocky on Friday September 23, 2022 @10:20AM (#32952305)
Sounds like a disaster in terms of potential bugs.
Re:The road to Zettalinux by jerf on Friday September 23, 2022 @10:50AM (#32952695)
It is, but it's the same disaster of bugs we already have from multiple independent cores sharing the same memory space, not a brand new disaster or anything.

It's a disaster of latency issues too, but it's not like that's surprising anyone either, and we already have NUMA on some multi-core systems which is the same problem.

We have existing tools that can be extended in straightforward ways to deal with these issues. And it's not like there's a silver bullet here; having separate address spaces everywhere comes with its own disaster of issues. Pick your poison.

Re:The road to Zettalinux by yencabulator on Friday September 23, 2022 @02:19PM (#32955357)
Just because you can refer to the identity of a thing anywhere in the cluster doesn't mean it can't also be memory-safe, capability-based, and just an RPC.
Re:The road to Zettalinux by lazide on Friday September 23, 2022 @08:54PM (#32958795)
Then why make said identity a fixed size number pretending to be a flat address space, instead of some other type of key?
Re:The road to Zettalinux by yencabulator on Saturday September 24, 2022 @02:42PM (#32965460)
For example, reads (when allowed) might just be transparent RAM accesses.

I'm not advocating 128-bit pointers, or saying they're useful or realistic. I'm just saying, what if.

Re:The road to Zettalinux by lazide on Saturday September 24, 2022 @03:11PM (#32965672)
The issue is latency. If you don’t want to care if the read or write will take 10ns or 10000ns, then that can work, assuming a flat permission structure too. Latency is a fundamental restriction for any computer that is > zero size, and the larger the ‘computer’ (data center), the more noticeable it is.

(For that matter, what happens when different segments of memory have complex access controls? What about needing to retry to failures like network partitions that don’t happen in a normal memory space?)

If latency matters, which it usually does, then you need some kind of memory access hierarchy, copying things back and forth, etc. and then you’ll almost certainly need a library of some kind to manage all this, prefetch from a slow range of memory and populate some of your fast local memory, etc.

At that point, we’ve done a lot of work, and are still pretending there is no network or the like, even though it’s there. It isn’t free, anyway. We’d also need checksums, cross network/fabric/access error handling, etc.

And with 128 bits, we could also use something like IPv6 with the lower 64 bits being byte address hah.

Re:The road to Zettalinux by yencabulator on Saturday September 24, 2022 @03:24PM (#32965783)
Data centers are already doing "disaggregated memory" (page out to remote RAM), as it's been faster than local disk for years now (or at least was in the pre-NVMe world).

The HPC world is all about RDMA to direct-access their huge data sets, and likely hyperscaler clouds are starting to do that too.

The latency gap is just treated as yet another layer of the cache model.

And we already have error correction for local RAM.

Re:The road to Zettalinux by lazide on Saturday September 24, 2022 @03:48PM (#32966006)
Which as you note, NVME and the latest PCI generations turns on it’s head.

Those were due to bottlenecks that don’t exist anymore, and even in your example were even then only emergency measures due to local resource shortages.

ECC is also not reliable/sufficient in the face of issues that arrive when networks start playing their part.

Re:The road to Zettalinux by rektide on Friday September 23, 2022 @11:44AM (#32953494)
Not that the industry doesnt broadly deserve this FUD take/takedown, but perhaps possibly maybe it might end up being really good & useful & clean & clear & lead to very high functioning very performant very highly observable systems, for some.

Having a single system image has many potential upsides, and understandability & reasonability are high among them.

Re:The road to Zettalinux by sidewndr46 on Friday September 23, 2022 @11:44AM (#32953491)
Instead of having another thread improperly manipulating your pointers and scribbling all over memory, now you can have an entire cluster of distributed machines doing it. This is a clear step forward.
Re:The road to Zettalinux by vlovich123 on Friday September 23, 2022 @10:42PM (#32959300)
I think the challenge is that within those pages you might have absolute pointers rather than offset to some “page” boundary. In that case, everything really must share a single uniform address space even if any given mode accessed only a small portion, no?

At that point, maybe you want 256bit or 512bit pointers so that you can build a single global addressable system for all memory in the world.

Re:The road to Zettalinux by maxwell86 on Friday September 23, 2022 @12:23PM (#32953975)
> How likely is it that any given node would be mapping out more than 2^64 bytes worth of virtual pages?

In the Grace Hopper whitepaper, NVIDIA says that they connect multiple nodes with a fabric that allows them to creat a virtual address space across all of them.

Re:The road to Zettalinux by rwmj on Friday September 23, 2022 @09:49AM (#32951938)
Distributed Shared Memory is a thing, but I'm not sure how widely it is used. I found that it gives you all the coordination problems of threads in symmetric multiprocessing but at a larger scale and with much slower synchronisation.

https://en.wikipedia.org/wiki/Distributed_shared_memory

Re:The road to Zettalinux by dragontamer on Friday September 23, 2022 @10:42AM (#32952582)
https://en.wikipedia.org/wiki/Remote_direct_memory_access

Again, I'm not a supercomputer programmer. But the whitepapers often discuss RDMA.

From my imagination, it sounds like any other "mmap". You, the programmer, just remembers that the mmap'd region is slower (since it is read/write to a Disk, rather than to RAM). Otherwise, you treat it "like RAM" from a programming perspective entirely for convenience sake.

As long as you know my_mmap_region->next = foobar(); is a slow I/O operation pretending to be memory, you're fine.

---------

Modern systems are converging upon this "single address space" programming model. PCIe 3.0 implements atomic operations and memory barriers, CXL is going to add cache-coherence over a remote / I/O interface. This means that all your memory_barriers / atomics / synchronization can be atomic-operations, and the OS will automatically translate these memory commands into the proper I/O level atomics/barriers to ensure proper synchronization.

This is all very new, only within the past few years. But I think its one of the most exciting things about modern computer design.

Yes, its slow. But its consistent and accurately modeled by all elements in the chain. Atomic-compare-and-swap over RDMA can allow for cache-coherent communications and synchronization over Ethernet, over GPUs, over CPUs, and any other accelerators sharing the same 64-bit memory space. Maybe not quite today, but soon.

This technology already exists for PCIe 3.0 CPU+GPUs synchronization from 8 years ago (Shared Virtual Memory). Its exciting to see it extend out into more I/O devices.

Re:The road to Zettalinux by c-linkage on Friday September 23, 2022 @03:09PM (#32955837)
Distributed Memory Access is just another kind of Non-Uniform Memory Access, which is Yet Another Leaky Abstraction. Specifically, if you care about performance at all you now have to worry about where in RAM your data lives.

Caring about where in memory your data lives is different from dealing with cache or paging. Programmers have to plan ahead to keep frequently accessed data in fast RAM, and infrequently accessed data in "slow" RAM. You'll probably need special APIs to allocate and manage memory in the different pools, not unlike the Address Windowing Extensions API in Microsoft Windows.

And once you extend "memory" outside the chassis, you'll have to design your application with the expectation that any memory access could fail because a network failure means the memory is no longer accessible.

If you only plan to deploy in a data center then maybe you can ignore pointer faults, but that is still a risk, especially if you decide to deploy something like Chaos Monkey to test your fault tolerance.

Re:The road to Zettalinux by zozbot234 on Friday September 23, 2022 @04:52PM (#32956907)
> And once you extend "memory" outside the chassis, you'll have to design your application with the expectation that any memory access could fail because a network failure means the memory is no longer accessible.

You have to deal with these things anyway in any kind of distributed setting. What this kind of location-independence via SSI really buys you is the ability to scale the exact same workloads down to a single cluster or even a single node when feasible, while keeping an efficient shared-memory programming model instead of doing slow explicit message passing. It seems like a pretty big simplification.

Re:The road to Zettalinux by jandrewrogers on Friday September 23, 2022 @06:06PM (#32957675)
I've written code for a few different large-scale SSI architectures. The shared-memory programming model is less efficient than explicit message passing in practice because it is much more difficult to optimize. The underlying infrastructure is essentially converged at this point, so performance mostly comes down to the usability of the programming model.

The marketing for SSI was that it was simple because programmers would not have to learn explicit message passing. Unfortunately, people that buy supercomputers tend to care about performance, so designing a supercomputer that is difficult to optimize misses the point. In real code, the only way to make them perform well was to layer topology-aware message passing on top of the shared memory model. At which point you should've just bought a message passing architecture.

There is only one type of large-scale SSI architecture that is able to somewhat maintain the illusion of uniform shared memory -- hardware latency-hiding e.g. barrel processors. If programmers have difficulty writing scalable code with message passing, then they definitely are going to struggle with this. These systems use a completely foreign programming paradigm that looks deceptively like vanilla C++. Exceptional efficient, and companies design new ones every few years, but without programmers that grok how to write optimal code they aren't much use.

Re:The road to Zettalinux by FuriouslyAdrift on Friday September 23, 2022 @03:25PM (#32956026)
RDMA is used heavily in SMB3 file systems for Microsoft HyperV failover clusters.
Re:The road to Zettalinux by throw10920 on Friday September 23, 2022 @10:49AM (#32952683)
That's what I was going to chime in with - you pay for that extra address width. Binary addition and multiplication latency is super-linear with regards to operand width. Larger pointers lead to more memory use, and memory access latency is non-constant with respect to size.

It might make sense for large distributed systems to move to a 128-bit architecture, but I don't see any reason for consumer devices, at least with current technology.

Re:The road to Zettalinux by dragontamer on Friday September 23, 2022 @10:51AM (#32952708)
> Binary addition ... is super-linear with regards to operand width

No its not. That's why Kogge-Stone's carry lookahead adder was such an amazing result. O(log(n)) latency with respect to operand width with O(n) total half-adders used.

It may seem like its super-linear. But the power of prefix-sums leads to a spectacular and elegant solution. Kogge-stone (and the concept of prefix-sums) is one of the most important parallel-programming / parallel-system results of the last 50 years. Dare I say it, its _THE_ most important parallel programming concept.

> multiplication latency

You could just... not implement 128-bit multiplication. Just support 128-bit pointers (aka: addition) and leave multiplication for 64-bits and below.

Re:The road to Zettalinux by meisanother on Friday September 23, 2022 @11:25AM (#32953185)
There's also something beautiful about seeing or creating a Kogge-Stone implementation on silicon.

I know it was one of the first time I thought to myself: this is not just a straightforward pipeline, yet it all follows such a beautifully geometrical interconnect pattern. Super fast, yet very elegant to layout.

Re:The road to Zettalinux by dragontamer on Friday September 23, 2022 @11:40AM (#32953408)
The original paper is a masterpiece to read as well, if you haven't read it.

"A Parallel Algorithm for the Efficient Solution of a General Class of Recurrence Equation", by Kogge and Stone.

It proves the result for _all_ associative operations (technically, a class slightly larger than associative. Kogge and Stone called this a "semi-associative" operation).

Re:The road to Zettalinux by meisanother on Friday September 23, 2022 @12:19PM (#32953927)
Well, just got it. Thanks for the reference!

A bit sad that 1974 papers are still behind a IEEE paywall...

Edit: Just finished reading it. I have to say that the generalization of 3.2 got a bit over me, but otherwise it's pretty amazing that they could define such a generalization. Intuition for those type of problem is often to proceed one step at a time, N times.

That it is provably doable in log2(N) is great, especially since it allows for a choice of the depth/number of processors you want to use for the problem. Hopefully next time I design a latency-constrained system I remember to look at that article

Re:The road to Zettalinux by dragontamer on Friday September 23, 2022 @02:42PM (#32955578)
> Hopefully next time I design a latency-constrained system I remember to look at that article

Nah. Your next step is to read "Data parallel algorithms" by Hillis and Steele, which starts to show how these principles can be applied to code. (Much higher-level, easier to follow, paper. From ACM too, so its free since its older than 2000)

Then you realize that all you're doing is following the steps towards "Map-reduce" and modern parallel code and just use Map Reduce / NVidia cub::scan / etc. etc. and all the modern stuff that is built from these fundamental concepts.

Kogge and Stone's paper sits at the root of it all though.

Re:The road to Zettalinux by meisanother on Friday September 23, 2022 @07:15PM (#32958185)
From Data Parallel algorithms by Hillis and Steele conclusion: 'if the number of lines of code is fixed and the amount of data is allowed to grow arbitrarily, then the ratio of code to data will necessarily approach zero. The parallelism to be gained by concurrently operating on multiple data elements will therefore be greater than the parallelism to be gained by concurrently executing lines of code'

I feel like this sums up the way of thinking one must have in this paradigm.

It's funny because when designing digital hardware, you're kind-of trained to see things under that angle, since often the commands you write will expand in that tree-like structure, but to gates/ALU instead of data.

Then managing the data-path often piggy-backs on the hardware structure you just generated.

I feel like I have a overall grasp on these concept, yet there's so much interesting results that I don't know about...

Re:The road to Zettalinux by throw10920 on Friday September 23, 2022 @11:40AM (#32953414)
You're right, binary addition isn't super-linear. It is non-constant, though, which is a slightly surprising result if you don't know much about hardware.
Re:The road to Zettalinux by dragontamer on Friday September 23, 2022 @11:44AM (#32953483)
Kogge-stone's O(Log2(n)) latency complexity might as well be constant. The difference between 64-bit and 128-bit is the difference between 6 and 7.

There's going to be no issues implementing a 128-bit adder. None at all.

Re:The road to Zettalinux by adgjlsfhk1 on Friday September 23, 2022 @11:50PM (#32959595)
a 10% latency hit on addition would definitely be noticable. that's roughly 1.5 years of IPC improvements at the current rate of CPU progress.
Re:The road to Zettalinux by dragontamer on Sunday September 25, 2022 @03:02PM (#32974794)
That's really not how that works.

Transistors have a "latency" measured in picoseconds, not nanoseconds.

A 128-bit adder will have a latency of 1-clock cycle, a 10% increase from a 64-bit adder of latency of 1-clock cycle. Aka: it doesn't matter cause it'd almost certainly fit inside of the clock cycle of modern CPUs.

Re:The road to Zettalinux by pclmulqdq on Friday September 23, 2022 @12:20PM (#32953947)
Multiplication of 128 bit numbers is also not a big issue. Today you can do it with 4 MULs and some addition, and it would still be faster than 64-bit division. Hardware multiplication of 128-bit numbers would have area problems more than speed problems. You could always throw out the top 128 bits of the result (since 128x128 has a 256 bit result), and the circuit wouldn't be much bigger than a full 64 bit multiplier.
Re:The road to Zettalinux by sakras on Friday September 23, 2022 @01:41PM (#32954935)
At least at this latest SIGMOD, it felt like everyone and their dog was researching databases in an RDMA environment… so I’d imagine this stuff hasn’t peaked in popularity.
Re:The road to Zettalinux by shadowofneptune on Friday September 23, 2022 @10:27AM (#32952393)
Even so, existing ISAs could address more memory using segmentation. AMD64 has a variant of long mode where the segment registers are re-enabled. For the special programs that need such a large space, far pointers wouldn't be that complicating.
Re:The road to Zettalinux by _the_inflator on Friday September 23, 2022 @11:01AM (#32952839)
I agree with you.

And seeing datacenter after datacenter shooting up like mushrooms, there might be some sort of abstraction running in this direction, that makes 128bit addresses feasible. At the moment 64bit seems like paging in this sense.

Re:The road to Zettalinux by eterevsky on Friday September 23, 2022 @10:51AM (#32952720)
8 EiB of data (in case we want addresses using signed integers) is around 20 metric tons of micro-SD cards one TB each (assuming they weigh 2g each). This could probably fit in a single shipping container.
Re:The road to Zettalinux by MR4D on Friday September 23, 2022 @12:01PM (#32953720)
But the cooling....

Seriously, that was great math you did there, and a neat way to think about volume. That's a standard shipping container [0], which is less than I thought it would be.

[0] - https://www.mobilemodularcontainers.com/products/storage-con...

Re:The road to Zettalinux by hnuser123456 on Friday September 23, 2022 @11:57AM (#32953675)
shipping container = 40x8x8ft

microsd card = 15x11x1mm, 0.5g

fits 437,503,976 cards = 379 EiB, costs $43.7B

219 metric tons

8 EiB ~ 10,000,000 TB = fills the shipping container 2.2% high or 56mm or 2 inches, 5 metric tons, costs $1B

shipping containers are rated for up to 24 metric tons, so ~40 EiB $5B 10 inches of cards etc

Re:The road to Zettalinux by bombcar on Friday September 23, 2022 @06:14PM (#32957735)
Don’t underestimate the bandwidth of a shipping container filled with SD cards.
Re:The road to Zettalinux by cmrdporcupine on Friday September 23, 2022 @09:43AM (#32951861)
Thanks for pointing out the `usize` ambiguity. It drives me nuts. I suspect it would make me even crazier if I was doing embedded with Rust right now and had to work with unsafe hardware pointers.

(They also fail to distinguish an equivalent of `off_t` out, too. Not that I think that would have the same bit width ambiguities. But it seems odd to refer to offsets by a 'size')

Re:The road to Zettalinux by torginus on Friday September 23, 2022 @02:14PM (#32955308)
And there's another disadvantage to 128-bit pointers - memory size and alignment. It would follow that each struct field would become 16 byte-aligned, and pointers would bloat up as well, leading to even more memory consumption, especially in languages that favor pointer-heavy structures.

This was a major counterargument against 64-bit x86, where the transition came out as a net zero in terms of performance, due to the hit of larger pointer sizes counterbalanced by ISA improvements such as more addressable registers.

Many people in high-performance circles advocate using 32-bit array indices opposed to pointers, to counteract the cache pollution effects.

Re:The road to Zettalinux by akira2501 on Friday September 23, 2022 @04:35PM (#32956756)
I figure the cache is going to be your largest disadvantage, and is the primary reason CPUs don't physically implement all address bits and why canonical addressing was required to get all this off the ground in the first place.
Re:The road to Zettalinux by api on Friday September 23, 2022 @09:46AM (#32951897)
I can't imagine a single Linux kernel instance or single program controlling that much bus-local RAM, but as you say there are other uses.

One use I can imagine is massively distributed computing where pointers can refer to things that are either local or remote. These could even map onto IPv6 addresses where the least significant 64 bits are a local machine pointer and the most significant 64 bits are the machine's /64. Of course the security aspect would have to be handled at the transport layer or this would have to be done on a private network. The latter would be more common since this would probably be a supercomputer thing.

Still... I wonder if this needs CPU support or just compiler support? Would you get that much more performance from having this in hardware?

I do really like how Rust has u128 native in the language. This permits a lot of nice things including efficient implementation of some cryptography and math stuff. C has irregular support for uint128_t but it's not really a first class citizen.

Re:The road to Zettalinux by gjvc on Friday September 23, 2022 @11:25AM (#32953183)
I can't imagine a single Linux kernel instance or single program controlling that much bus-local RAM, but as you say there are other uses.

MS-DOS and 640K ...

Re:The road to Zettalinux by api on Friday September 23, 2022 @11:44AM (#32953490)
The addressable size grows exponentially with more bits, not linearly. 2^64 is not twice as big as 2^32. It’s more than four billion times as big. 2^32 was only 65536 times as big as 2^16.

Going past 2^64 bytes of local high speed RAM becomes a physics problem. I won’t say never but it would not just be an evolutionary change from what we have and a processor that could perform useful computations on that much data would be equally nuts. Just moving that much data on a bus of today would take too long to be useful, let alone computing on it.

Re:The road to Zettalinux by gjvc on Monday September 26, 2022 @08:03AM (#32982049)
Thank you for the base-2 tutorial. I had no idea!
Re:The road to Zettalinux by dylan604 on Friday September 23, 2022 @11:39AM (#32953394)
>(comedy option: 32-bit int, 128-bit long, and 64-bit `unsigned middle`)

rather than unsigned middle, could we just call it malcom?

Re:The road to Zettalinux by TheCondor on Friday September 23, 2022 @05:10PM (#32957117)
It's very difficult to see normal computers that normal people use needing it any time soon, I agree. Frontier has 9.2PB of memory though, so that's 50bits for a petabyte and then 4 more bits, 54bits of memory addressability if we wanted to byte address it all. Looking at it that way, if super computers continue to be funded and grow like they have, we're getting shockingly close to 64bits of addressable memory.

I don't know that that really means we need 128bit, 80 or 96bits buys a lot of time, but it's probably worth a little bit of thought.

I don't know how many of you remember the pre-386 days. It was an effort to write interesting programs though, 512KB or 640KB of memory to work with but it was 16bit addressable and so you're writing code to manage segments and stuff, it's an extra degree of complexity and a pain to debug. 32bits seemed like a godsend when it happened. I imagine most of the dorks on here have ripped a blu-ray or transcoded a video image from somewhere, it's not super unusual to be dealing with a single file that cannot be represented as bytes with a 32bit pointer.

It's all about cost and value, 64bits is still a staggering amount of memory but if the protein folding problems and climate models and what have you need 80bits of memory to represent the problem space, I would hope that the people building those don't also have to worry about the memory "shoe boxing" problems of yesteryear too.

Re:The road to Zettalinux by green_on_black on Friday September 23, 2022 @10:23PM (#32959223)
Personally, having just suffered the issue of uint64_t (unsigned long vs unsigned long long...): I prefer having "bit size" + "semantics" in the type, and that's it. If the hardware doesn't support it, fail horrendously or use an approximation/slower path, as specified by compiler options.

In my opinion, developers should think about semantics first, optimization (device-specific or software) after (, unless you already know your device).

I slightly disagree with JVM "int = 32bit" but they essentially are forcing their own "virtual hardware", so I can understand that. For portable native code... I can only say that I'm disappointed in uint64_t. But also that, maybe, the current Rust isn't the end-all be-all of portable types.

Re:The road to Zettalinux by travisgriggs on Friday September 23, 2022 @10:42AM (#32952586)
> (comedy option: 32-bit int, 128-bit long, and 64-bit `unsigned middle`)

Or rather than keep moving the long goalpost, keep long at u64/i64 and add prolong(ed) for 128. Or we could keep long as the “nominal” register value, and introduce “short long” for 64. So many options.

Re:The road to Zettalinux by retrac on Friday September 23, 2022 @12:21PM (#32953952)
> 64 bits provides 16 EiB (16 x 1024 x 1024 x 1024 x 1 GiB), which is the sort of address space you might need for byte-level addressing of a warehouse full of high-density HDDs. Memory sizes don't grow like they used to

An exabyte was an absolutely incomprehensible amount of memory, once. Nearly as incomprehensible as 4 gigabytes seemed, at one time. But as you note, 64 bits of addressable data can fit into a single warehouse now.

Going by the historical rate of increase, $100 would buy about a petabyte of storage in 2040. Even presuming a major slowdown, we still start running into 64 bit addressing as a practical limit, perhaps sooner than you think.

Re:The road to Zettalinux by Spooky23 on Friday September 23, 2022 @11:10AM (#32952968)
Storage and interconnect specs are getting a lot faster. I could see a world where you treated an S3 scale storage system as a giant tiered addressable memory space. AS/400 systems sort of did something like that at a small scale.
Re:The road to Zettalinux by mort96 on Friday September 23, 2022 @11:18AM (#32953081)
We couldn't introduce a new 'middle' keyword, but could we say 'int' is 32 bit, 'long' is 128 bit and 'short long' is 64 bit..?
Re:The road to Zettalinux by musicale on Friday September 23, 2022 @07:15PM (#32958186)
> CHERI is a much more interesting case

It would be interesting to see something like this on x86 and ARM. I could imagine Apple implementing something similar.

Re:The road to Zettalinux by mhh__ on Friday September 23, 2022 @09:34AM (#32951736)
Systems for which pointers are not just integers have come and gone, sadly.

Many mainframes had function pointers which were more like a struct than a pointer.

Re:The road to Zettalinux by masklinn on Friday September 23, 2022 @09:38AM (#32951795)
Technically that’s still pretty common on the software side, that’s what tagged pointers are.

The ObjC/Swift runtime uses that for instance, the class pointer of an object also contains the refcount and a few flags.

Re:The road to Zettalinux by mhh__ on Friday September 23, 2022 @09:41AM (#32951835)
Pointer tagging is still spiritually an integer (i.e. mask off the bits you guarantee the tags are in via alignment) versus (say) the pointer being multiple addresses into different bits of different types of memory and tags insert other stuff here.
Re:The road to Zettalinux by masklinn on Friday September 23, 2022 @09:54AM (#32951993)
> Pointer tagging is still spiritually an integer

No, pointer tagging is spiritually a packed structure. That can be a simple union (as it is in ocaml IIRC) but it can be a lot more e.g. the objc “non-pointer isa” ended up with 5 flags (excluding raw isa discriminant) and two additional non-pointer members of 19 and 9 bits.

You mask things on and off to unpack the structure into component the system will accept. Nothing precludes using tagged pointers to discriminate between kinds and locations.

Re:The road to Zettalinux by apaprocki on Friday September 23, 2022 @09:49AM (#32951935)
Itanium function pointers worked like that, so it could have been the new normal if IA64 wasn’t so crazy on the whole.
Re:The road to Zettalinux by xxpor on Friday September 23, 2022 @11:58AM (#32953686)
A Raymond Chen post about the function pointer craziness: https://devblogs.microsoft.com/oldnewthing/20150731-00/?p=90...
Re:The road to Zettalinux by anotherhue on Friday September 23, 2022 @10:34AM (#32952478)
FYI Such memory tagging has a rich history https://en.wikipedia.org/wiki/Tagged_architecture
Re:The road to Zettalinux by aidenn0 on Friday September 23, 2022 @11:15AM (#32953046)
> ...which is the sort of address space you might need for byte-level addressing of a warehouse full of high-density HDDs

So if you want to mmap() files stored in your datacenter warehouse, maybe you do need it?

Re:The road to Zettalinux by tcoppi on Friday September 23, 2022 @12:03PM (#32953744)
Even assuming you are correct on all these points ASLR is still an important use case and the effective security of current 64-bit address spaces is low.
Re:The road to Zettalinux by lostmsu on Friday September 23, 2022 @09:58AM (#32952035)
So 64 bit address is only 1024 16TB HDDs? That number may go down quickly. There is a 100TB SSD already.
Re:The road to Zettalinux by ijlx on Friday September 23, 2022 @10:09AM (#32952159)
1024*1024 16TB HDDs, or 1,048,576.
Re:The road to Zettalinux by oofbey on Friday September 23, 2022 @10:09AM (#32952160)
1,000,000 drives at 16 TB each I think.

Kilo is 10 bits. Mega 20. Gigs 30. Tera 40. 16TB is 44 bits. 1000* is another 10 bits so 54.

Re:The road to Zettalinux by Beltiras on Friday September 23, 2022 @11:06AM (#32952914)
As my siblings are pointing out your error they are not addressing what you are saying. You are absolutely correct. It's conceivable within a few years to fit this much storage in one device (might be a full rack full of disks but still).
Re:The road to Zettalinux by ksec on Saturday September 24, 2022 @12:13PM (#32964022)
The highest density storage is currently 1PB in 1U, fitting 40PB in a Rack. We expect to reach 100PB in 2025, with some futuristic roadmap of 200PB in 2030, assuming we could yield those NAND. And a thousand layers NAND to reach 500PB in 2035.

That is still an order of magnitude smaller than 16EiB. So no, not in a few years time.

Re:The road to Zettalinux by guenthert on Saturday September 24, 2022 @09:04PM (#32968034)
One thing is storage density, another is data transfer rate.

Frontier's LustreFS based mind-boggling 700PB Orion storage subsystem writes at a similarly impressive 5TB/s [1]; so with a generous reading, it could potentially fill up in about 39h. Historically, disk density increased more rapidly than network bandwidth, which ultimately limits the performance of distributed file systems.

If it takes weeks or months to write such enormous data set, then there will be only few use cases, if it takes years there will be none.

[1] https://www.ornl.gov/news/frontier-supercomputer-debuts-worl...

Re:The road to Zettalinux by guenthert on Sunday September 25, 2022 @01:36PM (#32973772)
> network bandwidth, which ultimately limits the performance of distributed file systems.

Oops, that's nonsense. I had my coffee now...

Re:The road to Zettalinux by infinityio on Friday September 23, 2022 @10:11AM (#32952193)
Not quite - 1024GiB is 1TiB, so it's 1024 x 1024 x 16TiB drives
Re:The road to Zettalinux by lostmsu on Friday September 23, 2022 @12:49PM (#32954279)
Ah, parent already corrected their mistake. The comment I was responding to was saying 16*1024*1024*1GB.