Idiomatic python is slow. Trying to convert everything into the mini-DSL that a C implemented library can understand is oftentimes considerably faster.

In case of most ordinary datasets it simply doesn’t matter in practice. Even comenius logo would be adequately fast.

> almost always crystal clear and self explanatory

and slow. Most of the time the idiomatic pandas answer takes advantage of its vectorization capabilities while plain python would require nested loops.

This is one of the advantages of languages with optimizing compilers. In Python, there's a tradeoff between clarity and performance. But in C++ for example, your for-loop is just as fast as any library's for-loop.

Edit: My assumption is that many Python libraries are written in C for performance, but maybe that's not the case.

>But in C++ for example, your for-loop is just as fast as any library's for-loop.

This is most definitely not true. Even stuff like subtle aliasing guarantees can make code run an order of magnitude faster or slower.

Another example is the order in which you access memory. To multiply two matrices, in one you move by row and in the other you move by columns. One makes the cache happy and the other unhappy, so it's faster to transpose one of them (the correct one, I never remember) and them "multiply" them. We notice this in Fortran with -O2, but the difference disappears in Fortran with -O3, so the compiler is doing some magic.

In case of these libraries it is mostly the case of painstaking optimizations in low level code. No common language has sufficiently smart auto-vectorization to reliably convert for loops in most cases to SIMD instructions - so no, your average C++ for loop wouldn’t be that much faster, unless you spend a lot of time optimizing it for the specific task.

Except that your C loop can be WRITTEN slower than some library’s loop.

Someone pointed out to me recently as you say, the idiomatic python sometimes seems like more of a barrier. It takes more time to parse and grok, for their human brain, a for loop with a zip having multiple interables vs nested for loops. I've been thinking about this. So take, a list comprehension, supposedly better than a for loop at times. The canted take would be that list comprehensions are said in a way of what a human wants, the for loop is more how to do it, and the former is better ipso facto. And I think I've come to appreciate this difference and preference.

My conclusion is that I rather like the opinionated take of python. It might take a liiiiittle bit of a learning curve, but you do come to appreciate the zen of choices they've arrived at after a few decades of going at it.

Worth pointing out that list comprehensions aren't just a stylistic preference or syntactic sugar, they're significantly better performance than the equivalent for loops.