forcing pretty much all computation no matter how trivial
   into contortions via numpy incantations so that the inner
   loops can run in native extensions

Welcome to the development of Matlab-style code. It can be pretty disheartening to see a complicated hand-crafted algorithm replaced by, say, a matrix multiplication. Try to find Matlab/numpy implementations from paper authors. This programming style is a bit difficult to get used to initially. But it will become incredibly powerful as you start thinking in algebraic operations.

Also, don't do debug runs on the full data. Implement ideas with a subset that computes fast. Then simplify the code and make it fast. Then run it on the full dataset.

If you absolutely need to combine linear algebra operations with fast imperative code, have a go at the Julia language [1] or use a C++ library like Eigen [2] or Armadillo [3].

[1] https://julialang.org/

[2] http://eigen.tuxfamily.org/index.php?title=Main_Page

[3] http://arma.sourceforge.net/

In a practical sense, the libraries that make up the Python scientific stack -- numpy, pandas, matplolib, sklearn, etc. -- are all akin to domain-specific languages, each with its own magic behavior, quirks, and recommended best practices that one must master. It takes a long while to become proficient in all of them. The good news is that there's a large, growing community of helpful people using these tools, so you can usually find the answer to any question with a single Google search.

For numpy specifically, if you are interested in getting the most out of it with the fewest possible headaches, I would recommend the following online book: http://www.labri.fr/perso/nrougier/from-python-to-numpy/

Otherwise, please make sure to read bmarkovic's spot-on comment in this thread: https://news.ycombinator.com/item?id=14158597

From a stack-overflow (https://stackoverflow.com/questions/13432800/does-performanc...)... "I remember I read somewhere that performance penalty is <1%, don't remember where. A rough estimate with some basic functions in OpenCV shows a worst-case penalty of <4%".

I'm working on a project that downloads a 2 hour video, frame by frame. Then after, I run image detection on a photo or multiple photos against every single one of those frames.

My code was slow as shit at first. I decided to profile and realized the only speed hog was SIFT's detectAndCompute(). It was taking ~.5s for each frame passed to the algorithm.

So I ended up trading memory for speed and now create huge PyTables loaded with every single frame's KeyPoints and Descriptors. I do this when I first download the videos frames and even though that takes a while, I can now run image detection against 6000 frames (100 minutes) with however many photos in about (24 minutes).

Point is, even just using the built-in cProfile helped a lot because there was only one function that was truly affecting my python. Other than that, I love the python translation. I'm surprised you have memory problems though. OpenCV always murdered my CPU's but never was really memory intensive

The reason why python is the right tool for this job is because it has such libraries for just about everything, from serial communications to a bunch of hardware driving stepper motors, relays and reading inputs to imaging and number crunching and all the other bits that went into this. I'd have a much harder time achieving the same effect in any other language and likely it would have cost me a lot more time.

Python is not perfect (far from it) but it gets the job done.

I appreciate this comment as well -- and indeed I will agree that my expectations are part of the problem.

I'm not actually doing 'performance critical' work -- but rather prototyping tasks -- the speed of my feedback cycle is the only thing that's performance sensitive within the context of my rant -- and this includes time to run the code and the time to debug it.

I should also acknowledge other (self-induced) problems including the need to run code on a remote server for extra hardware capability vs my pitiful laptop.

I should acknowledge that part of my frustration could come from my development process as from the inherent nature of the underlying tools themselves ... I've experimented with my process -- trying to find something that works for me and I've ended up with this hobbled kludge of tools:

- a script that re-runs rsync on file changes - atom + hydrogen extension connected via ssh to a jupyter kernel_gateway - some ssh terminals where I run longer running code from command line - an sshfs mount of a remote directory on the server for viewing some output artifacts ...

Avoidable runtime errors after longish-running processing tasks are a very frustrating time-sync ...

1. If you are prototyping and prefer working on a remote server, have a look at(https://notebooks.azure.com/), memory is limited to 4Gb but its free and jupyter notebooks are great for prototyping.

2. For prototyping you should try and do REPL driven development, what I mean is that you should be ok with just playing around with the library API before you write a longish-running processing task, that would reduce(not remove) the chances of a runtime error. Jupyter notebooks excel in this too as you can just try out code in a cell, learn from it, rinse and repeat. You can also use your IDE to send code fragments to REPL and get immediate feedback on them. This way of iterative development would make sure that the speed of your feedback cycle isn't slow. If python's feedback cycle seems slower to you than C++, you are definitely not using the REPL enough :)

3. If debugging is a pain point definitely give an IDE a try, I prefer Visual Studio as I am on Windows. You can very well go with Pycharm or vscode(not technically an IDE but has a debugger so that's that). I personally prefer Jupyter notebook or Emacs with my code and REPL in split windows for prototyping, different strokes for different folks.

I personally love working with python, I even write my blog posts in a Jupyter Notebook so although it isn't perfect, it doesn't have to be frustrating either :) Hope my suggestions can be of some help!

Python frequently saunters into the second territory. Well, I guess there are tools available to profile memory usage and stuff, but if I had to spend that much effort tracking down memory issues, I might as well rewrite it in C++. (It doesn't help (or it helps?) that I'm much more comfortable with C++ than Python. YMMV.)

Take this gem from a well known course:

   np.concatenate([x.next() for i in range(x.nb)])

I think we need a better abstract language for checking matrix algebra operations.

Option 1: F# or C# with the Deedle[1] library. Deedle provides series and dataframe classes, along with various stats functions. I believe that there are also some vis tools. Type providers in F# allow you to specify a data source, such as a CSV or database, and then not only infer the types but also give you intellisense autocompletion. See the F# guide for data science[2] for more info.

Option 1.5: I hesitate to recommend the following, because there's simply not much here yet, but there is a dataframe library[3] for Nim[4]. Nim is a strongly typed language with a Python-like syntax which compiles to C and is apparently quite fast. It has multiple options for garbage collection but also supports manual memory management. It offers lisp-like macros for implementing DSLs, which the dataframe library I mentioned uses quite a bit. The main problems with Nim are of course the lack of libraries and the need for a notebook-like environment such as Jupyter, which are certainly big problems indeed. But I think that Nim is something to look out for over the next few years.

As much as I like Deedle and F#'s features, I've personally decided to abandon the use of Microsoft technologies due to their many user-unfriendly actions regarding Windows 10 and privacy. I don't fault anyone else for using Deedle, though, because it is a nice tool. This is just a personal decision of mine.

[1] http://bluemountaincapital.github.io/Deedle/ [2] http://fsharp.org/guides/data-science/ [3] https://github.com/bluenote10/NimData [4] https://nim-lang.org

Python IS strongly typed. It is also dynamically typed.

As for myself, the first serious language I taught myself was Python, so I'll always have a soft spot for it. Now that I've begun to realize Python's shortcomings, I've started to work more with statically typed languages; my favorites are Scala and C++, and my next goal is to work with Rust.

[0] https://pypi.python.org/pypi/pudb

Personally, I love Python. It's been working fairly well for our ML projects utilizing scikit-learn to process fairly large spatiotemporal datasets (temporally varying 2D and 3D datasets). I find numpy to be critical for keeping runtimes reasonable.

Regardless of the language, profiling may be necessary in order to obtain acceptable performance.

Also, Python has plenty of abstraction capabilities, just like C++. There's no reason all your code should be throwaway. Make use of classes, magic methods, list comprehensions, hashes, the Pandas library (lots of powerful abstractions for managing data in there).

PyCharm professional edition comes with a pretty nifty profiler.

Now, I would kill for the ability to compare two profiles, but putting them on two different tabs will do for now ...

Python is very popular for the use cases you describe, but I have long felt it's akin to using a Honda Civic that has been souped up to be a specialized race car. You can definitely make a Honda fast around a racetrack, but that's not what they were designed to do. You're better off using a purpose built machine.

That being said, a purpose built racecar is harder to drive, doesn't have AC, needs more maintenance, has more expensive tires that wear out faster, etc...

Nothing in life is free, there is always a trade off.

With that in mind, you can try to program your numerics code hiding as much as possible the fact that you use numpy. I have found that this is the only barely bearable way to do numerics/image processing in python.

In [3]: x = np.array([1,2,3])

In [4]: x[0:500] Out[4]: array([1, 2, 3])

However, after translating some exercices from Octave to python/numpy, I have experienced a great deal of frustration. A simple three-line solution in Octave becomes a fifteen-line behemoth that does not even work consistently across different versions of the system.

It sounds like you're mostly having trouble with OpenCV, which has a horrid API, especially the python version adds another layer of confusion on top of it, and dumping things back and forth from opencv Mat objects to numpy arrays sucks and wastes time. The documentation also is fairly bad. The error messages, like you say, kinda suck. So I can't help you much with that but know that the opencv python bindings fall WAY below the standard of what's considered good. Stuff like simplecv is a bit better but doesn't cover nearly as much as OpenCV does.

Numpy itself is great though, for how little code you often need for a complex, vectorized matrix operation. I've used stuff like Eigen (or even boost matrices or raw BLAS stuff) which I think is way more painful to deal with. For the annoying cruft that any of those bring, the minor speed boost over numpy isn't worth it for me most of the time.

The other thing is, when folks complain about the slowness of python when doing computationally heavy stuff like this, it's often the situation that they have a bunch of vectorized operations, and in the middle somewhere, they stick in something like a regular python for loop, which is orders of magnitude slower. The trick, especially for hot loops, is to just avoid using anything but vectorized operations. In something like C++ of course you're less likely to notice this happening because there is much less overhead, but there'd still be an impact of doing stuff the non-vectorized way anyway.

For profiling, take a look at this: https://www.huyng.com/posts/python-performance-analysis plus the internal cprofile module that does call-graph profiling. It's nothing amazing like vtune or whatever, but it gets the job done.

As far as architecture goes - the relative lack of typing structure in python definitely makes life easier in many ways and cuts down on boilerplate, but it also makes it much easier to shoot yourself in the foot like you say. One thing to keep in mind is that when writing data heavy code there is an additional layer of concerns in play. Stuff like the "shape" of your data. It sounds like if you're running into this, what you want is to make a big diagram / graph of how data is flowing through your code, each stage of computation, and what the data looks like before and after. When you understand this well, it's much easier to reason about the code.

One thing that helps to know is that the style of programming in the python world is a bit different. In C++ land, the instinct is to start with building architecture, and when you inevitably realize that you've misunderstood the problem domain, you do a massive refactor. This usually constitutes large changes in how your code is structured but C++ gives you tools, like strong types, to do this rather easily. With python, unless you're setting of on a massive project, you start with minimal architecture. Write code that does the thing you need. Then, as you find yourself repeating yourself or doing unnecessary things, or dealing with complexity you don't need to know, only then do you start giving structure to your code bit by bit, as you need it. It's a different philosophy that's the result of a different set of functionalities.

In the larger context, I mostly have the opposite experience as you - I can pull out C++ whenever I need it, but I almost never do. The set of tools I have for computation an analysis in python are so much richer. For example, imagine c++ code for reading an excel sheet, and scraping some data off the web, cleaning both, joining them together, running a complex mathematical calculation, and saving a few plots. And I can do this from scratch in half an hour, around 100 lines of code. I also have an interactive shell that I can explore the data and the libraries I have, and I'm not stuck in the code-compile-code-compile loop. The productivity boost is amazing. In contrast, I don't even want to imagine what it would be like to attempt to do this in C++. Python is great glue. Admittedly, you're pushing at the edges of what python is useful for. It's definitely possible (I've written computer vision stuff with python before, and it worked OK), but if you're not making use of any of these benefits of using python, you might as well use C++ and enjoy the speed and strong typing.

Hope this helps your python experience be a bit smoother!

Also, do you use pandas, and why not?