Designing file systems for newer storage devices is an engineering problem that requires a lot of effort, but not a great amount of new insight.
Designing a new programming language that somehow exposes the concepts you mentioned more transparently while still being actually useful would require major, major breakthroughs and insights in PL design.
I think a larger problem with new architectures is that consumer adoption follows price/performance/power, not any inherent architectural quality. The architecture we're stuck with is the one that most hardware devices get sold with (x86/ARM right now). That gets determined by hardware OEMs, which in turn make their decisions based on what'll help them sell the fastest devices with the lowest power consumption for the least money. So something like RISC V is fascinating and quite elegant, but until there are RISC V chips that are cheaper and faster than Intel ones, it remains an academic curiosity. Then if you're a compiler writer, you gotta work with what you've got for an installed base, and you can't really get adoption for a new language unless it lets startup founders unlock new markets because your combination of development velocity + execution speed lets them do things they wouldn't otherwise be able to.
1.) automatically defining SoA/AoS transformations for data types in the language, and
2.) automatic profiling of code execution counts & array lengths to determine when inserting such a transformation would save more time than the transformation costs.
3.) native SIMD primitive types
4.) GPU backend a la Futhark
5.) native syntax for parallel execution, where the compiler enforces no data sharing (or read-only shared data) between executions.
6.) [more speculative] pluggable backend for this syntax so that you could run these concurrent executions sequentially, using SIMD instructions on the same thread, in multiple threads, on the GPU, or spread across multiple machines in a cluster without altering the code. That'd let you scale up from a few K to a few T without altering your code.
7.) I/O mechanisms that default to mmap, 0-copy vectored I/O, or RDMA as appropriate.
8.) An unfold primitive in either the language or standard library that lets you compute multiple values with a single traversal over a list, ideally while still keeping each individual computation isolated and modular. I find this is a really common pattern in my Kotlin code, and the biggest culprit for still writing manual accumulator loops.
9.) Support for multi-pass initialization in data structures, where you can hold some fields as uninitialized for a later computation, but mark the whole structure as immutable once every field has been computed.
10.) A standard library that uses these parallelization techniques under the hood so you get really good speed for common operations but it just works. For example, Boyer-Moore can be done on the GPU. Many common regexps can be reduced to string search on a common left prefix + a DFA on potential matches, where the prefix search can be done with Boyer-Moore. That opens the possibility of blazingly fast search over gigabytes of data for common regexps like /<a href=["']?([^"'\s])["']?>(.+?)<\/a>/. Many times the algorithms you'd use when your haystack is ~1G are very different from the algorithms you'd use when your haystack is ~1K, which goes back to the utility of an integrated profiler.
It occurs to me that a lot of these boil down to the inefficiency of doing lots of traversals of arrays in a world where touching memory is usually the critical path. So for example, this is slow but modular:
val high = oneBillionFloats.max()
val low = oneBillionFloats.min()
val total = oneBillionFloats.sum()
val average = oneBillionFloats.sum() / oneBillionFloats.size()
var high = Float.MIN_VALUE
var low = Float.MAX_VALUE
var total = 0
var count = 0
oneBillionFloats.forEach {
high = maxOf(high, it)
low = minOf(low, it)
total += it
++count
}
val average = total / count
The whole thing reminds me about the disputed articles, "Are U.S. Railroad Gauges Based on Roman Chariots?" and by extension the Space Shuttle was limited by the width of two horses walking side-by-side. I don't think software is all that different than other standards like switching to the metric system, or Southern California switching their power standard from 50hz in 1948 to match 60hz (Japan never unified their power). It's a huge investment and a lot of it is political appetite for adopting the change.
It seems pretty obvious that the average person doesn't grok/like using folders (everything goes on the desktop). I was excited when Microsoft and Apple seemed experimental in the late 90's early 2000's with WinFS and Smart Folders. That kind of fizzled out. I think the reliability prevents adoption and it's a bit of a chicken and the egg with architecture vs presentation. Gmail supporting tags instead of folders seemed to drive everyone nuts for awhile (it still seems to come up when exposing it to other clients).
Like a Ship of Theseus...
It's weird that basic filesystems today are so out of touch with modern realities that we are universally forced to resort to using complex databases even in cases when the logical model of files and directories fits the storage needs really well.
It's weird that hierarchical storage is the only universal model available on all OSes and in all languages.
The more I think about it, the more I realize that we live in a bizarro world where software runs everything, yet makes little to no sense from either human, or modern hardware or system design perspectives.