I think Rob is right to call out the problem, but is being a bit rose colored about Plan 9.

Plan 9 was definitely ahead of its time, but it's also a far cry from the sort of distributed OS we need today. "Everything is a remote posix file" ends up being a really bad abstraction for distributed computing. What people are doing today with warehouse scale clusters indeed has a ton of layers of crap in there, and I think it's obvious to yern for sweeping that away. But there's no chance you could do that with P9 as it was designed.

"Everything is a file" originally referred to read and write as universal object interfaces. It's similar to Smalltalk's send/receive as an idealized model for object-based programming. Hierarchical filesystem namespaces for object enumeration and acquisition is tangential, though it often works well because most namespaces (DNS, etc) tend to be hierarchical. (POSIX filesystem semantics doesn't really figure into Plan 9 except, perhaps, incidentally.) Filesystem namespacing isn't quite as abstract, though (open, readdir, etc, are much more concrete interfaces), making impedance mismatch more likely.

The abstraction is sound. We ended up with TCP and HTTP instead of IL and 9P (and at scale, URLs instead of file descriptors), because of trust issues, but that's not surprising. Ultimately the interface of read/write sits squarely in the middle of all of them, and most others. To build a distributed system with different primitives at the core, for example, send/receive, requires creating significantly stronger constraints on usage and implementation environments. People do that all the time, but in practice they do so by building atop the file interface model. That's what makes the "everything is a file" model so powerful--it's an interoperability sweet spot; an axis around which you can expect most large-scale architectures to revolve around at their core, even if the read/write abstraction isn't visible at the point users (e.g. application developers) interact with the architecture.

A hierarchical namespace is fine, but the open/read/write/sync/close protocol on byte based files is definitely inadequate. The constraints on usage you decry are in fact fundamental constraints of distributed computing that are at odds with the filesystem abstraction. And this is exactly what I was getting at in talking about rose colored glasses with P9. It in no way is a replacement for something like Colossus or Spanner.

> P9 ... in no way is a replacement for something like Colossus or Spanner.

Colossus and Spanner are both proprietary so there's very limited info on them, but both seem to be built for very specialized goals and constraints. So, not really on the same level as a general system interface like 9P, which is most readily comparable to, e.g. HTTP. In Plan 9, 9P servers are routunely used to locally wrap connections to such exotic systems. You can even require the file system interface locally exposed by 9P to be endowed with extra semantics, e.g. via special messages written to a 'control' file. So any level of compatibility or lack thereof with simple *nix bytestreams can be supported.

Or proper graphics programming as well.

This describe more a Single System Image [0] to me (WPD includes Plan 9 as one but considering it does not does not supports process migration I find it moot). LinuxPMI [1] seems to be a good idea but they seems to be based on Linux 2.6, so you would have to heavily patch newer kernel. The only thing that seems to support process migration with current software / still active are CRIU [2] (which doesn't support graphical/wayland programs) and DragonflyBSD [3] (in their own words very basic).

[0]: https://en.wikipedia.org/wiki/Single_system_image [1]: http://linuxpmi.org [2]: criu.org [3]: https://man.dragonflybsd.org/?command=sys_checkpoint&section...

I don't really see any reason to consider process migration a required feature of either a distributed os or a single system image. Even on a single computer this isn't always practical or desireable (ie. you can't 'migrate' a program running on your gpu to your cpu, and you can't trivially migrate a thread from one process to another either).

Not all units of computation are interchangeable, and a system that recognizes this and doesn't try to shoehorn everything down to the lowest common denominator actually gains some expressive power over a uniform system (else we would not have threads).

> his describe more a Single System Image [0] to me

No, Plan 9 is not a SSI OS. The idea is all resources are exposed via a single unified file oriented protocol: 9p. All devices are files which means all communication happens over fd's meaning you look at your computer like a patch bay of resources, all communicated with via read() and write(). e.g.:

  [physical disk]<-->[kernel: sd(3)]-----< /dev/sdE0/
  [audio card] <---->[kernel: audio(3)]--< /dev/audio
  [keyboard]-------->[kernel: kbd(3)]----< /dev/kbd

Looking above it looks like Unix but with MAJOR differences. First off the disk is a directory containing partitions which are just files who's size is the partitions size. You can read or write those files as you please. Since the kernel only cares about exposing hardware as files, the file system on a partition needs to be translated to 9p. We do this with a program that is a file server which interprets e.g. a fat32 fs and serves it via 9p (dossrv(4)). Your disk based file system is just a user-space program.

And since files are the interface you can bind over them to replace them with a different service like mixfs(4). /dev/audio is like the old linux oss where only one program could open a sound card at a time. To remedy this on plan 9 you run mixfs which opens /dev/audio and then binds itself over /dev replacing /dev/audio in that namespace with a multiplexed /dev/audio from mixfs. Now you start your window manager and the children programs will see mixfs's /dev/audio instead of the kernel /dev/audio. Your programs can now play audio simultaneously without changing ANYTHING. Now compare that simplicity to the trash fire linux audio has been and continues to be with yet another audio subsystem.

Keyboard keymaps are a filter program sitting between /dev/kbd and your program. All it does is read in key codes and maps key presses according to a key map which is just a file with key->mapping lines. Again, keyboards are files so a user space file server can be a keyboard such as a GUI keyboard that binds itself over /dev/kbd.

Now all those files can be exported or imported to other machines, regardless of CPU architecture.

Unix is an OS built on top of a single machine. Plan 9 is a Unix built on top of a network. It's the closest I can get to computing nirvana where all my resources are available from any machine with simple commands that are part of the base OS which is tiny compared to the rest.

Does Plan 9 have an equivalent to alsa or jackaudio or pipewire, where you can pick a (usually shared memory) ring buffer size, have the kernel or daemon alert you (usually through a poll fd) when there's room in the buffer, and only then perform the necessary calculations (instead of performing a blocking write which adds latency)? What about synchronously receiving blocks of input audio as soon as available, and outputting equally-sized aligned blocks of output audio played back exactly two periods of buffering after the matching input audio? Or chaining three or more apps in a pipeline, all woken in sequence whenever input data is available, and still have only two periods of latency?

This is not supported yet but if you'd like to have a go at it, patches welcome.

Best explanation of Plan 9 I've ever seen

Graphical programs could be checkpointed and restored as long as they don't directly connect to the hardware. (Because the checkpoint/restore system has no idea how to grab the hardware's relevant state or replicate it on restore.) This means running those apps in a hardware-independent way (e.g. using a separate Wayland instance that connects to the system one), but aside from that it ought to be usable.

For CRIU it is not supported: https://criu.org/Integration#Wayland.2FWeston, also in my experience it doesn't work. Are you talking about an other software?

It has been done "virtually" by going through e.g. VNC https://criu.org/VNC . Alternately, CRIU apps could be required to use virt-* devices, which CRIU might checkpoint and restore similar to VM's.

For what it's worth, the HPC-standard way of checkpointing/migrating distributed execution (in userspace, unlike CRIU) is https://dmtcp.sourceforge.io/ It supports X via VNC -- I've never tried -- but I guess you could use xpra.

If you yearn for Plan 9 -- I'm not sure I do -- Minnich's current incarnation of the inspiration seems to be https://github.com/u-root/cpu

Meh. Every time a 9p server dies, every client dies. Plan9 is not comparable to k8s.

Are there any good walkthroughs of what a good, distributed plan 9 setup looks like from either a development or a administration perspective? Particularly an emphasis on many distributed compute nodes (or cpu servers in plan 9 parlance).

Glad to see Plan 9 getting some love in the comments even if it didn't make it into the article.