Forget the word "compilers" for a second. Pick the most powerful language you can find for experimenting with graph algorithms.
That's it. That's all it boils down to. Compiler bloat, running speed, start up time and other stuff is just a programmer's wishful thinking. How many machines in the world run your software? how much of your user's time have you wasted by not implementing a piece of software as efficiently as you could? (would your users trade delivery time for startup or running time? i.e. do they want to start using your program today, even if it runs 300% slower than the machine allows, or do they want to use it next week and expect it to run at maximum speed? :-)
Comparing ML to Perl or Python is ridiculous. There are a number of native optimizing compilers of ML. ML is closer to C++ or Java or CL performance-wise than to those languages.
Writing it in ML may be nice, but nobody wants to install ML just to get at another language. The kind of person that would do this is probably aspiring to a good language implementation of their own that could be considered a /peer/ of ML (or Perl, or Python, or Scheme, or Java, or....). High level languages written in yet another high level language are more of an academic curiosity than a tool anyone's likely to use.
While, yes, you're talking about prototyping, that assumes doing so in ML is much simpler. Maybe for you, but not for a C/C++ coder who doesn't know ML. There are a very, very large number of such coders.
Why write a prototype then have to rewrite the whole thing in C to have a "serious" language implementation? For small languages, the effort spent in prototyping then rewriting is going to be more effort than just writing a good stand-alone implementation from the outset. My hobby "prototype" compilers are small, portable, fast, depend on almost nothing, and integrate with the system and command line nicely. If the language was to become useful embedded in some program, say, I could do so immediately because I wrote the implementations to be good from the start. As a user, I'd take that over something I have to run in ML or some other massive high level language any day...
Would it be nice if the system's language was something better (somewhat) like ML? Sure. Is it? No.
Anyway, from the implementation point of view, when it comes time to write your GC, or any other low level details (parallelism?) writing in something like ML is a severe handicap. As far as performance goes, LLVM makes /fast/ code, and a lot of work is constantly being poured into it by a lot of people to make it even faster. This is a HUGE amount of work (i.e. far beyond what one person writing a language, or the people involved in the projects you have mentioned, can or will do), and it's already been done for you if you use LLVM (of course LLVM does have ML bindings).
Anyway, the point is, if you're trying to "prototype" or do PLT research, and you know ML well, and all you care about is the language itself you're implementing (and not the qualities/bloat/dependencies/performance of the implementation); sure, using ML probably makes sense. However these things are certainly not true of everyone writing a compiler.