If you are not being specific it is impossible to know what you are talking about. It could be a library thing or it could be that your are really just describing the initial latency of the JIT compilation.

If you look at the well known language shootout you will see Julia today beats almost everything even on multicore. The clear exception is anything related to garbage collection. But you can circumvent this with turning off the garbage collector temporarily.