Curious student here, would you mind elaborating a bit on this unexplored territory?

This is a bit like asking "Can you elaborate on particle physics" in 1900. The point is that we don't know because there's been so little experimentation in the space. Not a lot of funding for "build an OS that uses completely different idioms and concepts to what we know, possibly without even a goal other than trying something out".

Smalltalk vms, forth, Oberon, lisp machines, amoeba etc

Here's a HN comment from earlier this week that elaborates on some earlier operating systems that explored some really novel ideas: https://news.ycombinator.com/item?id=33768592

To summarize:

> ITS (of PDP-10 hacker fame) - processes could debug and introspect their child processes. The debugger was always available, basically. The operating system provided support for breakpoints, single-stepping, examining process memory, etc.

> KeyKOS (developed by Tymshare for their commercial computing services in the 1970s) - A capability operating system. If everything in UNIX was a file, then everything in KeyKOS was a memory page, and capabilities (keys) to access those pages.

In every operating system, the basic unit of abstraction will be a process -- which necessitates a scheduler, some form of memory protection, some way for the process to interact with the kernel, and the notions of "kernel space" and "user space". There is a lot of room for innovation there (see ITS), but I suspect most of the room for innovation is in how an OS abstracts/refers to various parts of the system.

Possible, but apparently rarely worth the extra effort or complexity to think about.

The Unix ‘everything is a file’ has done well because it works pretty well.

It also isn’t generally a security issue, because it allows application of the nature things we use for files (ACLs, permissions, etc), without having to come up with some new bespoke idea.

You do realize that RTKit is not a Unix-like OS, right?