I was doing hand-optimizing of 6502 on the Apple ][+ back then. Well, a little bit, admittedly. The thing to do seemed to be a mix of BASIC and machine language. CALL-151 for the win!
I admit I was way off - Windows has only been around since 1985 or so, and in usable form since 1990. And Java does have some very fast code. And I agree about optimisations too. It's easy to go overboard to get more speed.
I also believe the Java stack has some very carefully optimized code too. It certainly has the performance to back up that affirmation. Besides that, it was designed to be easily portable to other OSs and architectures, a feature that's more important, IMHO, than speed under Windows.
Finally, you should only optimize for speed the code your profiler says has big impact on your performance. The rest you should optimize for readability and correctness. There is a lot of very subtle bugs we introduce when trying to optimize stuff and it's much easier to optimize correct code than to fix blazingly fast, heavily optimized buggy code.