News for hackers. Stuff that matters.
Re:Using D and std.ndslice as a Numpy Replacement by jboy on Saturday January 02, 2016 @10:58PM (#10829433)
Credit to the D developers for providing a concise, carefully-designed library for N-D array processing. The chained method invocations demonstrate D's UFCS (Uniform Function Call Syntax) nicely. And it's a definite bonus that you can use underscore like a comma separator in long integer literals (eg, `100_000`).

But if you use Python + Numpy/Scipy/Matplotlib and you're looking for a modern, compiled language for execution speedups or greater flexibility than what Numpy broadcasting operations provide by default, I would recommend Nim. It's as fast as C++ or D, it has Pythonic syntax, and it already includes many of D's best features (including type inference, UFCS, and underscores in integer literals).

And best of all, you don't need to rewrite all your existing Python+Numpy code into a new language to start using Nim.

The Pymod library we've created allows you to write Nim functions, compile them as standard CPython extension modules, and simply drop them into your existing Python code: https://github.com/jboy/nim-pymod

The Pymod library even includes a type `ptr PyArrayObject` that provides native Nim access to Numpy ndarrays via the Numpy C-API [ https://github.com/jboy/nim-pymod#pyarrayobject-type ]. So you can bounce back and forth between your Python code and your Nim code for the cost of a Python extension module function call. All of Numpy, Scipy & Matplotlib are still available to you in Python, in addition to statically-typed C++-like iterators in Nim+Pymod [ https://github.com/jboy/nim-pymod#pyarrayiter-types , https://github.com/jboy/nim-pymod#pyarrayiter-loop-idioms ]. The Nim for-loops will be compiled to C code that the C compiler can then auto-vectorize.

Re:Using D and std.ndslice as a Numpy Replacement by 9il on Sunday January 03, 2016 @12:38AM (#10829657)
Re:Using D and std.ndslice as a Numpy Replacement by jboy on Sunday January 03, 2016 @01:02AM (#10829713)
It looks like you need to copy your D array to a newly-allocated Numpy ndarray before you can pass it to Python. So there's no binary PyArrayObject interoperability between D & Python (right?). Copying large N-D arrays all the time sounds slow...

(That Matplotlib example uses the function `d_to_python_numpy_ndarray` in the PyD project, which I found defined here: https://github.com/ariovistus/pyd/blob/master/infrastructure... . It clearly allocates a new Numpy array: https://github.com/ariovistus/pyd/blob/master/infrastructure... )

Also, I couldn't find any examples of invoking D functions from Python. In fact, I could only find mentions on the D mailing list of people reporting that they couldn't get it to work: http://forum.dlang.org/post/[email protected]...

By compiling (transpiling) to C, Nim really does have an unfair advantage in the interoperability challenge...

Re:Using D and std.ndslice as a Numpy Replacement by 9il on Sunday January 03, 2016 @02:10AM (#10829819)
ndslice was merged to DLang master repo today. It is not a problem to fix PyD. (compiling to C is crispy)

EDIT: Exposing-d-functions-to-python http://pyd.readthedocs.org/en/latest/functions.html#exposing...

Re:Using D and std.ndslice as a Numpy Replacement by stevieboy on Sunday January 03, 2016 @01:39AM (#10829779)
jboy, Can nim-pymod be used as VLA's for nim? I'm not too fond of the nim seq'type (bit slow for my usage) and prefer arrays, but need their length allocated at runtime. Can this be done via (albeit a clunky route) through nim-pymod? i.e arrays created and accessed all in nim (no python)?
Re:Using D and std.ndslice as a Numpy Replacement by jboy on Sunday January 03, 2016 @02:05AM (#10829816)
Hi, the short answer is "Yes, but ...".

Yes, Pymod's PyArrayObject can be created & accessed entirely in Nim; yes, its length (actually, shape) is specified at runtime; and yes, it can be resized after creation.

However, Pymod's PyArrayObject is designed for maximum binary compatibility with Numpy. As such:

1. It uses Numpy's array creation functions (such as `createSimpleNew`), which in turn uses the Python runtime memory allocator. So at the very least, Pymod assumes you've linked against `-lpythonX.Y`.

2. It integrates with Python's GC rather than Nim's GC. (This is why it's passed around as `ptr PyArrayObject` rather than `ref PyArrayObject` -- it's untouchable by the Nim GC, but instead uses Python refcounts.)

Of course, we do intend to extend Pymod to include a pure-Nim sibling for PyArrayObject, but it doesn't exist yet.

If you want a high-quality no-Python-dependency VLA in Nim right now, I'd recommend this library by Andrea Ferretti [ https://github.com/unicredit/linear-algebra ]. It can link against your system BLAS, in addition to offering GPU support using NVIDIA CUDA. Andrea is another member of the Nim community who does a lot of scientific computing: http://rnduja.github.io/2015/10/21/scientific-nim/

Re:Using D and std.ndslice as a Numpy Replacement by stevieboy on Sunday January 03, 2016 @02:23AM (#10829832)
Ok, looking at the linear-algebra VLA stuff now. Thanks for the detailed reply and suggestion. (and apologies to original OP for veering off-topic... the new ndslice package looks like a pretty cool addition the D libraries, I'll be having a look at that too)