You can't compare starting up a statically-typed, compiled Swift repl to those other dynamic, interpreted languages -- it's doing a lot of things that running a normal, compiled Swift program wouldn't do, and that dynamically-typed Clojure repl doesn't do (even though it's slower).

The Swift repl is essentially a debugger running a compile-run cycle on each entered expression. Some statically-compiled languages have slow compilers but extremely fast runtime execution (Rust, and to a lesser extent C++, come to mind).