See Amdahl's law http://en.wikipedia.org/wiki/Amdahl%27s_law
Even a tiny amount of work spent sequentially (eg a scheduler or gc) can determine performance as the core count ramps up.
Erlang is essentially ahead of almost everyone on minimalising the sequential portion.
Even discounting specialized hardware+VM solutions like Azul, the regular JVM runs fine on shipping hardware that has lots of cores.
The dominating limiter in practice is actually garbage collection. I.e., very large heaps will drive the scale up vs. out decision way before cores is a serious issue.