If you are running Clojure you need to run at least clojure.core. Loading that is handled - surprise - by the JVM. As would be expected, if you run other implementations (like Dalvik), this can change dramatically (for worse, usually).
So it is correct that loading the JVM itself is not what's taking most of the time. But loading take so much time because of the JVM.
Here's a pretty good (if a bit outdated) analysis on Clojure startup time: http://clojure-goes-fast.com/blog/clojures-slow-start/
The question is, could Clojure be rewritten in a way that loads significantly faster with the same behavior on the JVM? I'm not sure.
And especially on big projects, the lack of DCE on the JVM can lead to loading a ton of useless code that significantly impacts startup time outside of the Clojure language.
So there are two parts to the solution: The first part is Small Clojure Interpreter (sci) which is a "(subset of) Clojure written in Clojure", so kind of McCarthy for Clojure. A cool project in itself but it still needs a VM to run.
So here is where the second part comes in: the GraalVM produces native binaries from various things, including Clojure code and SCI is simple enough to work with GraalVM.
It's cool how these two pieces of tech combine together to create something greater than the sum of the parts.