You could tell people "download Firefox to use my spiffy 3D site" (or download Chrome or whatever) just like people now tell them "download Flash Viewer 9". Does Papervision in Flash take advantage of 3-D acceleration hardware? In those ways I'm not sure there's so much of a difference between JavaScript in browsers and ActionScript in Flash.

Graphics-specific optimizations are probably going to happen, if by that you mean "fixing Canvas so it isn't painfully slow".