I personally use Mathematica for this purpose and find it indispensable. I wonder if the PL designers should focus more on this class of languages, instead of features for production languages (like fancy type systems).
In Julia, explorative programming is productionized programming, you don't have to switch to a different or restricted set of tools to make your program fast, it's the same code that you can profile on the fly. With types, you can patch your hotspots by adding a specific method to the generic function, and it will get invoked automatically instead of the more general case algorithm.
In those languages, I can actually take the theoretical practical hardware limits (memory BW, throughput, latency), and relatively easily achieve 99% of their utilization.
When people say “very fast” and “my cores are at 100%” they often mean “my program achieves 1% of the perf the hardware can deliver”.
So? That's the case for every JIT or interpreted language...
Dragging something in from scikit-learn was literally like...two lines of code. It's not the same...investment as most FFI systems.
So ~2x faster in Julia? Though it's a sort of awkward comparison, since this type of iteration seems a little anti-idiomatic in Python/pandas and doesn't fully leverage type information in Julia.
I do most of my programming in Julia and Python. Python libraries like numpy and pandas are fast and efficient—if you're staying "within the lines" of how the library is designed to work. And most of the time this is okay! But not irregularly I want to do array operations on arrays of user-defined types, or I want to walk through a dataframe row by row without paying a huge performance penalty, etc. And all the sudden the well-tuned Python ecosystem feels very restrictive.
In Julia, my workflow is roughly: 1) think about the problem. 2) Code an intuitive solution. 3) If necessary, tweak a little bit of code to improve performance by reducing allocations or type instability.
That's a lot less mental work than my Python/Matlab/Java workflow of 1) think about the problem. 2) Think about how the solution can be expressed in the paradigm the language supports performance with. 3) Write a solution in this particular paradigm. 4) Tune for performance, which may be awkward if the initial solution was not intuitive.