You might want to take a look at e.g., Scala and how it implements actors on the JVM.
It's basically doing a MxN actor to thread-pool scheduling.
The dispacter is probably a bottleneck, looking at the paper http://lamp.epfl.ch/~phaller/doc/haller07actorsunify.pdf I can't really tell how it's implemented.
How does the Scala run queue work? eg have a look at the Erlang article I linked to in my first post.
The Erlang way of handling concurrency is to spawn millions of "processes" (threads) obviously these can't be hardware threads so Erlang is hitting problems OS schedulers would handle (or not be able to handle) in the JVM's case.
I would argue that the JVM is eventually going to have to handle the threading itself if it's to support any actor based programming. OS schedulers have other uses they need to be fast for and hardware threads will always have higher overhead. Python's Stackless is perhaps an example of the amount of work it would require.