Files exist because persistent storage is slow, copying data from RAM to disk is slow and we'd like to have a way to refer by name to blobs of bytes, but in the end it is only there because it's a useful metaphor. If the metaphor gets in the way of our ability to use the storage medium you can either do a mapping (like the direct memory mapping you refer to), which in this case would almost be like a backwards compatibility layer, a more involved scheme where all RAM is backed by persistent storage and devices only keep a flag which objects are currently being worked on and which are stable (an 'open' file would simply be marked as currently in use by an application but there would be no explicit save, just an unlock), or a completely new system based on an image like architecture (see: smalltalk, some lisp implementations). There are some drawbacks to that model (such as harder collaborative development) but I see no reason that applications could not be packaged that way even if the development used a more traditional method.

It would be fairly trivial to layer a conventional filesystem on top of such a persistent object store to get the best of both worlds, which would - amongst others - give you instantaneous suspend and re-awaken and other goodies.

If true this will vastly change the way we use computers.

This was the original architecture of Multics BTW -- the world just consisted of a single address space of segments (pages) plus a capability-based address space; a higher level construct gave you a named structure for a group of segments.

A similar approach has been taken with HP's interesting memristor-based "Machine"

There's a lot of interesting stuff in Organick's book on Multics, not all of which was implemented, unfortunately. And a lot of really good stuff was tossed overboard when fitting Unix into a PDP-7 (a lot of overwrought bad stuff was jettisoned too -- don't get the wrong idea!).

This is a great question. One way I imagine it:

Short term, the disk controller becomes a peripheral on the memory bus. On a 64-bit x86-64 system, the top 16 bits of the address are either 0000 or ffff for RAM. Make it so that the prefix 1000 (for example) maps to the disk, so accessing (physical) address 1000000013371000 accesses byte 13371000 on the disk.

Now processes can just ask the OS to perform a physical memory mapping to obtain a range of virtual addresses directly backed by disk pages, with page protections set based on their filesystem permissions. Such physical address mapping interfaces already exists in most OSes to support memory mapped I/O (for example, mapping /dev/mem in Linux).

This addressing scheme has another advantage: other devices on the system can use e.g. DMA to directly talk to the disk without any CPU intervention. For example, the GPU could load textures straight off of disk, just like John Carmack wants.

Medium term, we start rethinking the filesystem. If we make the address range for a given disk completely persistent, we can just put pointers to disk bytes on the disk itself. Processes will use the same virtual addresses as the physical addresses when talking to the disk. Suddenly "serialization" to disk is no longer required: data structures can be stored in native form directly on the disk. Imagine having a "dmalloc" function call hand you a chunk of persistent storage which you treat the same as any memory, but which can outlive the process. Similar concepts exist in some languages (like MUMPS), and now we bring the idea to all programming environments.

Long term, RAM ceases to be an independent entity, and merely becomes OS-managed cache for the big persistent storage (assuming it still has any latency/bandwidth advantages by this point). Now you can get rid of the notion of "shutting down" or "starting up" the system: everything is persistent. Without having to constantly refresh DRAM to keep the system alive, devices can "sleep/hibernate" more frequently and readily, saving significant power. Programming models become nearly unrecognizable as old models of memory management and process lifetimes give way to new models of persistent storage management and eternal services.

We're not far off from seeing a potential revolution in computing here.

>Suddenly "serialization" to disk is no longer required

That's not true. You'd still want excel to just save your sheets as xls file and show it to a coworker as opposed to sharing the entire excel program state. Upgrading from one version to another also requires a stable persistent data format for things such as configurations.

Not to mention that you still need serialization to communicate over a network.

Yes, you're definitely right - "serialization" as a concept is going to still be needed for data interchange. I just didn't want to mention it because that's just status quo, not part of the new computing model.

In my original comment, I simply meant that you would no longer have to serialize your data to your own disk.

That's almost a weird amalgam of really old school computer architectures and new.

It seemed to me from the layperson descriptions of how the old tube-based systems were operated, some of the old architectures had no notion of files at all, it was all RAM all the time. I also read stories of when rotating media first started being used, programmers wrote data not sequentially on the media but in an optimized pattern so by the time the cpu pulled a value into RAM the read head was positioned multiple units over, and so data was written out according to the demands of the hardware, and now we're talking about taking the cpu out of most of the loop.

We aren't coming full circle with the new memory architectures we're exploring, but kind of a spiral in a "history doesn't repeat but rhymes" manner by dusting off some old techniques and putting new spins on them, pun not intended.

If always-persistent state becomes popular, then I wonder if that will revive the popularity of image-based environments, a la Smalltalk and Lisp Machines, along with the collaboration challenges those introduced.

http://pmem.io - for byte addressable form factors

http://spdk.io - for block device form factors

Memory addressable non-volatile storage instead of SSDs that process block storage read/write commands, and file-system improvements to allow direct mmap of file contents from NVM into your process address space without copying through RAM.

I'm curious, why do you say that the open/read/write model can't work? I suppose that current software should be optimized to take advantage of better performance, but I don't see why it is obsolete.

10ns latency means that a file entry/inode/block is available just 10-30 CPU cycles after it is requested. Suddenly the 1000 CPU cycles spent dispatching the system call look very inefficient by comparison.

Obsolete might not be the right term, but there is essentially no way that open/read/write can take proper advantage of this kind of low latency storage. We are going to need a new way to abstract and interface with this persistent storage.

Read/write are certainly a bit slow, but this may well be acceptable for most applications. High performance apps can use mmap, which can be extended to directly map the nonvolatile memory in question.

Or a way to make system calls faster, some sort of vdso with the ability to switch privilege level.

That's how __vsyscall works now.

__vsyscall simply abstracts away the actual system call invocation strategy (int, sysenter, syscall), but doesn't change the model at all.

Anyway, what I was thinking was similar to call gates, which were phased out for being slow. Probably just making syscall faster in the CPU and reducing the overhead kernel side would be enough.

What's wrong with plain old mmap ?

mmap() tricks you into thinking you can directly access the disk. What really happens when you access the mapped data is that the CPU generates a page fault, the OS takes control, copies the data from disk to physical memory and maps the page in your address space. So essentially it is a convient way to do read().

The mmap() model is good. The current page fault mechanism, not so much.

The difference is that you only take the page fault once per page (or less than once if you do some prefetching in the kernel), while you have to do a system call for every read().

It costs a fortune to map a file on every OS I've tried, it makes little sense unless you're consuming a lot of data from few files rather than little data from many files

The main reason to keep it are: 1) network 2) what if we uncover a faster cpu/mem ? We would need to reintroduce file-systems. I'd say it's safe to deprecate it after about 30 years after it become obsolete. (So we are sure nothing will surpass it any time soon.)

For now. If you build it they will come. Things like these take time but they won't happen if the hardware guys don't move first.

>Consumer OS I/O software stacks are completely unable to deal with this. It was fine for spinning drive latency, it's okay(ish) for high-performance SSD but the open/read/write model just cannot work that fast. It's several orders of magnitude off.

Citation needed.