The best 3d web engine is Unity3D. Best game engine tool is Flash, even for some small 3D.
You can always have a downloadable client but if you want good web based browser games Unity3D and Flash (AS3 since it is on AVM2 and much faster) are the only way to go.
If you want a pretty sweet Python 3D engine check out Panda 3D.
http://www.realchat.com/blog/java-vs-flash/
By the way I am not discounting Java as a viable platform. It works in many cases, Runescape proves that being the #2 North American MMO. But it doesn't do that with graphical quality. It does it with gameplay and the progression etc. As of right now the best looking platforms and easiest to develop for are Unity3D and Flash for the web, if we are talking developing games for the browser.
EDIT: btw those numbers are old, these are newer: http://kaioa.com/node/40 But still 1.6 is about 70% penetration.
Flash loses out largely for the lack of good accelerated 3D. I've played with papervision but it's just too slow for a anything decent, that I can guarantee will run on even moderately equipped machines.
Despite what people say, Java plugin penetration is still very high and games like Runescape show what you can do. The ability to run cross-platform OpenGL in the browser is a huge advantage.