Java wouldn't work very well!
In particular, running a program from bash would simply classload it into the existing VM, not invoke a whole separate VM, and then it'd be more or less instant, except you'd have the potential for much more flexible APIs and combinations of tools. Look at PowerShell for an example.
Powershell is a terrible example for the performance point I am trying to make, whatever about its flexibility.
[0] http://www.javalobby.org/java/forums/t72620.html
[0] http://stackoverflow.com/a/13496610
[1]http://www.ibm.com/developerworks/library/j-multitenant-java... (this now looks less experimental but I'm willing to bet >1 instance of eclipse, or bash for that matter would not work very well).
I'm not sure how this works: http://www.excelsiorjet.com/
but it might help to alleviate the problem even though it still has a jvm.
https://en.wikipedia.org/wiki/Java_Classloader
looking at what you mention now, can a program classload itself? seems not: A class with a given name can only be loaded once by a given classloader.
I doubt you would even have to go as far as native extensions for GUI before you start running into problems even though the programs are written in managed code.
Can you better explain how in practice Java programs could share a VM in replacing a typical Unix bash environment/userland?
Would they have to use the special IBM JVM? Would bash have to contain grep as a class? e.g.
machine:~$ grep include Source | grep -v 32 | grep -v 16
Reading about "JAR hell" I really don't think it would work very well.
Interestingly in the IBM link provided earlier they load up substantial non GUI servers such as Tomcat, Jetty and JRuby and achieve a startup time that is twice as fast.
They also have hello world:
Hello World Print "HelloWorld" and then sleep
Multi-tenant JVM: 309
Hand-tuned: 73
Default: 63
Improvement with multitenant: 4.2X to 4.9X
Even with this I think a C++ version would eat it for breakfast. It would be interesting to find out what would happen if a Single JVM were loaded on boot, all programs were loaded into that and compare the JVM based Unix on those terms with e.g. Solaris or something.