This idea was implemented in EROS, and we've been exploring it for Robigalia as well. Storage is a finite resource which can be transferred between processes, including an "auction" mechanism which allows two processes to examine a trade before agreeing to it.

Doesn't this already exist for processor scheduling?

There's a weighting in many such systems, but ultimately it's still just a queue, usually a FIFO one.

Niceness allows for higher-priority processes to preempt others, but doesn't address the problem of an overwhelmed queue.

And processor scheduling isn't memory allocation. Time is ultimately some percentage of wall-clock (and/or overcommittment). Memory is ... different.

There's also the question of such stuff as garbage collection and scheduling of that. I had the opportunity to do some JVM tuning "ergonomics" (horrible name) a few years back. Turns out that you get far better behaviour in most cases by decreasing the sweep frequency and increasing the the allocation chunks (terminology is escaping me), due to the fact that natural attrition deallocates memory, and running sweeps too frequently simply chews up massive amounts of CPU time with no return on freed memory.

We also identified processes which genuinely did require very large memory allocations, and allocated hardware specific to those.

Specific workflow and process understanding (always idiosyncratic to a particular work assignment) was necessary, and took time to acquire.