Worry about -Xms rather than -Xmx for startup times.

I suspect there are ways to make it faster (e.g. there's a nashorn runner that starts up appreciably faster than "proper" java), but the main use case for java is server-side programs that run for days or weeks between restarts, so it's optimized for that use case.