If you do stick with the JVM, it tends to trade memory for speed, unless you tune it otherwise. For startup speed, start it once, connect a REPL, and keep it running. A started REPL on a JVM instance can be flushed very quickly when you need a blank slate.
But yes, slow starts are annoying if you're starting a JVM frequently. Can you change your workflow so that's not needed? Check out Component, Mount or Integrant.
We're using Integrant already so yes, I can mostly work in a REPL without restarting. But the restarts are more like 15 seconds. I'll have to measure and see what's so slow. My rdev platform is Cursive on a 16G MBP.
And while I'm commenting, yes, we have a reason to run on JVM (a lot of legacy Java code we probably want to interface with later on). So thanks to other commenters for suggesting Node.js and Common Lisp, but not possible. :)
Just a recommendation if you want to avoid the JVM bloat issues, which are real and have existed since the JVM was developed
peace