While Numpy/Scipy is great with a capital G, it seems the underlying infrastructure is flaky.
I'm talking about the messy process of building and installing it. And it seems everything is held with duct-tape
Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed.
Most any significant scientific package I've ever had to build (which is quite a few) had a complex build system that felt like it was built on top of shaky glass pillars. It's just the nature of what's involved and it's a fundamentally hard problem; it's difficult to replace those libraries because the extreme amount of verification/validation and optimization that have gone into something like LAPACK makes it economically undesirable.
I'd say also that this is part of the appeal of NumPy and SciPy, which is they glue together a huge number of tools that are otherwise disjoint. You don't have to worry (most of the time) about matrix ordering, calling conventions, those mysterious "workspace" variables in LAPACK etc.
That and Cristoph Gohlke[0]'s repository of pre-compiled modules. That guy saved me more than once. Should I ever meet him, I'll sure buy him a beer!
If you're using Python for scientific computing, you can use the Anaconda distribution of Python, it makes it painless to work with those packages.
> "Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed."
Python has excellent ORMs, I'm not sure what you're getting at.
% pip install numpy
For more (e.g., other OS'es, build-from-source), see:
My C code was very complicated in order to be fast. Lots of pointer swapping, bit-shifting, etc.
The code was about 4% faster in C, which was more lines of code, much harder to read (and write), compiled with every imaginable flag that could offer a speedup. Switching operating systems, this code was about 8% faster in C on Linux than Python in Windows 10. C in Linux would take about 9.5 days to complete, Python in Windows 10 would be about 10.5 days.
In addition, Python has the "pickle" module so that I can easily make the algorithm save state to pause and restore it any time I want. What's more, I can probably use mmap to share memory easily without making copies and switch to multiprocessing for an even greater speedup (my first attempt using Pool was a little slower despite using 3 cores, but that's the slowest way to multithread).
Plus, with each variable being ~10Mb, I was concerned about memory copying (I studied mechanical engineering, I missed a lot of this stuff in school), and with C I could make memory copying vs passing pointers explicit. All of that was unnecessary.
It was another reinforcement of the idea: "Build it first, then optimize".
Python in science is kinda like the One Ring the Binds them - it is the glue that holds together the HPC code at CERN, ESA, NASA, and more, it crunches climate models and delivers your daily weather forcasts. Some really great projects like Cython, Theano, and Numba boost the performance of numeric code to native speeds with a fraction of the hassle of hand-written C. Mountains of machine learning research is done in Python, as well as statistics and analytics. And of course Numpy remains the de-facto standard of N-dimensional array libraries - if you look at numeric computing libraries in many other languages they will be framed as "Numpy-for-X".