Are you aware that Hotspot was born as a Smalltalk (StrongTalk actually) JIT compiler?
Other than that I _think_ maybe ST has a fixed number of slots per class compared to dynamically added instance/class variables in ruby? My memory is fuzzy.
By the way, there _is_ a ruby built by smalltalk developers[0] exactly because the object model is 99% the same.
http://lambda-the-ultimate.org/node/2606
Ruby permits adding methods to individual objects; in Smalltalk, all methods reside in classes.
In Ruby, it is practical and somewhat useful to add methods dynamically; in Smalltalk, the practice is generally to treat the methods and classes as static.
Have also a look at:
http://www.hokstad.com/the-problem-with-compiling-ruby
I don't think all those 5 problems apply equally well to Smalltalk, do they?
All Ruby methods reside in classes, too, "adding a method to an object" is really adding it to the object's singleton class (metaclass).
The difference between Ruby and Smalltalk is that in Smalltalk, each explicit class has an implicit metaclass (a class in which the class's instance methods are defined), whereas for Ruby each object (including, but not limited to, class objects -- and notably including other implicit metaclasses) has an implicit metaclass (often called the eigenclass).
In effect, from an implementation point of view (at least for MRI), they're just normal classes except for when you define them, and a handful of insignificant cases where you have to check a flag to determine whether or not to consider them.
I wrote that one in frustration when working on my Ruby compiler project, actually, and I think there are reasonable solutions to all of them, and in fact you will find that most of them, possibly all are solved to various extents in either Smalltalk implementations or Self implementations. In particular the Self papers is an invaluable resource for anyone who wants to implement languages as dynamic as Ruby, or Javascript...
The main problem is the effort. And implementations like MRI really has not tried very hard to address any of this - you'll note it first got a bytecode version (instead of AST walking) with 1.9, and still does not JIT. AFAIK they've not even tested simple optimizations like partial vtables to avoid the costly method dispatch etc.
A big part of the challenge is that Ruby is a remarkably hard language to implement, not just or even mainly because of the dynamic features - those make it hard to make it fast, not hard to implement - but because it's still largely unspecified, the parsing is chock full of annoying corner cases, and there are lots of dark, hairy areas of Ruby that almost nobody ever see. I love Ruby, but I think it badly needs a revision that deprecates a whole chunk of functionality that nobody really uses but that affects implementations, as well as writing a proper spec and then tighten up a whole lot of areas where users would be largely unaffected, but that would make it vastly easier to implement.
So a lot of effort that could have gone into making implementations faster, go into addressing annoying digressions.
Besides just nitty-gritty optimizations there are some high level design issues which are interesting. JVM performance is great for languages like Java, but for some reason probably related to the JVM all Java web servers seem to be multi-threaded monsters, which turns debugging into a mess and seems to make DoS attacks way easier. We use a self-hosted Jira on top of Tomcat at work, there are maybe 30 simultaneous users and every time one of them does something computationally expensive or uses some buggy functionality the whole process goes nuts, memory leaks until exhaustion, load goes to maximum etc., the whole web sever has to be restarted. With a web server with multiple worker processes you just restart the single worker and call it a day. Does anyone happen to know why the worker-based hosting model with separate worker processes seems to be completely absent from the Java world in favour of multi-threaded servers?