I used to work for a vendor, and I've written a significant fraction of such an environment.
> so that for example you can't store your code in files and easily connect them with bits of code in another language
Which is why I know this to be patently false. You can actually add compiler objects and have select methods written in SQL or whatever language you like.
> Essentially, your programs would have to become image-based
Again, patently false for the REPL use case. Also there are a number of declarative Smalltalks already, which don't need to be started from an image file. Also, Smalltalk MT used to compile down to a DLL indistinguishable from any C++ DLL.
But incidentally, do you happen to know offhand of any resources about adding new compilers like you were talking about? That sounds pretty neat. I unfortunately missed the golden age of Smalltalk, so most of what I know comes from playing around with Squeak and GNU Smalltalk and reading about older implementations.
The examples for the T-Gen compiler compiler in VisualWorks Smalltalk talked a little about this, and had a toy SQL example. IIRC, there was a <pragma: > you could put in a method to use your own Compiler object. Also, Smalltalk/X actually let you implement methods in C right in the Smalltalk class browser.
Unfortunately, a lot of documentation and other info are lost in the mists of time.
I'd guess "the image thing" is probably the reason Smalltalk hasn't taken off more than it has.