In [5]: %timeit pickle.dumps(a)
1 loops, best of 3: 724 ms per loop
In [6]: %timeit pickle.dumps(a, protocol=pickle.HIGHEST_PROTOCOL)
100 loops, best of 3: 12.4 ms per loop(That is my humorous way of saying that Python 2.3 is rather old and therefore HIGHEST_PROTOCOL seems like it would be a desirable trade off for most.)
You might look into trying to port the django libs by yourself if you have the skill to do so. 2to3 often gets you really far. If you don't, I'd definitely recommend opening a ticket on the project page. I've done that with a few libs I use, and for a couple of them the maintainers just totally forgot about them and got around to doing the conversion just because I asked.
How do you use MyPy? I'm particularly worried about two things:
1) If I'm building a library, I can't have MyPy as a requirement, but would still like to use it for the types checks. Is there a way to omit the import when distributing your library?
2) Can it only check parts of an application? Maybe I have a big Django app and don't want it to static-check Django and all the other imports every time, for example.
1. No, afaik there's no way to omit the import. Sadly there's little in the way of macros or preprocessors in the python world at this point.
2. I'm fairly sure there's ways to use module stubs, but all my mypy work has dealt with the stdlib which presents no issue.
%timeit p = pickle.dumps(a)
1 loops, best of 3: 1.14 s per loop
But tostring and fromstring run in a fraction of the time. %timeit s = a.tostring()
1000 loops, best of 3: 1.13 ms per loop
%timeit b = np.fromstring(s).reshape(800, 600, 3)
1000 loops, best of 3: 1.57 ms per loop
np.all(a == b)
Truehttps://github.com/eishay/jvm-serializers/wiki
Although they're mostly limited to Java, I know at least one (Avro) can be used in Python.
Thanks!
You'd allocate memory in the parent process and then you'd write to that memory from within the child processes. The Python function you're "multiprocessing" won't necessarily need to return anything.
It would be good if this could use a nanomsg and its zero-copy mechanism instead of just raw sockets.
I work with video data, so I just ran these rough benchmarks on my MacBook Pro:
It takes nearly a second to serialize a not-so-huge `numpy` array, which makes it very difficult to do any sort of soft real-time analysis.This is a huge pain, and (very sadly for this Python aficionado) suggests that Python might be the wrong language for this kind of work.
Any suggestions?