News for hackers. Stuff that matters.
Re:The NumPy array: a structure for efficient numerical comp... by agibsonccc on Monday August 01, 2016 @03:06AM (#12200658)
The numpy array is a great data structure. I know it's not python but I wanted to plug our attempt at this for the jvm, nd4j:

http://nd4j.org/tensor

We have a java wrapper:

https://github.com/deeplearning4j/nd4j

Scala:

https://github.com/deeplearning4j/nd4s

And underlying c++:

https://github.com/deeplearning4j/libnd4j

We heavily dog food it in our deep learning library but we're hoping this can be more broadly useful for people.

Re:The NumPy array: a structure for efficient numerical comp... by hatmatrix on Monday August 01, 2016 @07:06AM (#12201484)
That's great but what about nd4j's data structures resemble numpy arrays? Without making connection to the original most this comment just reads like typical spam.
Re:The NumPy array: a structure for efficient numerical comp... by jnbiche on Monday August 01, 2016 @07:44AM (#12201628)
In my experience, it's perfectly acceptable on HN to plug a related project in a comment, as long as it's not done repeatedly for totally unrelated projects (in 5+ years here, I can only think of 1 or 2 examples of this actually happening on a frequent basis).

Given that these are open source and not even commercial projects, and that this is presumably an enthusiastic developer for those projects just sharing his/her work with a potentially interested audience, I hardly think the "spam" label is appropriate.

And to answer your question specifically, based on what I've just read, ND4J offers an optimized n-dimensional array intended for numeric/scientific computing, very much in the manner of numpy. This is presumably what the comment's "our attempt at this for the jvm" refers to.

Re:The NumPy array: a structure for efficient numerical comp... by agibsonccc on Monday August 01, 2016 @08:29AM (#12201824)
Yes it's a port of the associated concepts to the jvm. FWIW: we actually do use this commercially ;). It's apache though so do what you want with it etc.

Apologies if it came off like that but the main intent was to show an alternative implementation. If you actually look at the nd4j.org link you'll also notice a simpler/more concise description of the same concepts.

Eg: linear buffers,shapes, strides, offsets, dimensions

Re:The NumPy array: a structure for efficient numerical comp... by jnbiche on Monday August 01, 2016 @11:08AM (#12202923)
Apologies for assuming it wasn't commercial just because it was open source, my mistake. In any case, in my experience, comments regarding related non-commercial and commercial projects are quite welcome on HN.

It also looks like you have some support for GPU acceleration, which is cool since as far as I know (barring recent developments), numpy doesn't have this natively.

Re:The NumPy array: a structure for efficient numerical comp... by agibsonccc on Monday August 01, 2016 @11:17AM (#12203009)
Oh haha not at all! You're right most of the time.

Yes we put a lot of work in to multi gpu and the like. Send me an email if you're interested in any details! Emails in profile.

Re:The NumPy array: a structure for efficient numerical comp... by jnbiche on Monday August 01, 2016 @12:17PM (#12203609)
I'll go ahead and ask a question here so others can benefit: do you have any plans to support OpenCL, or will you focus on Cuda?
Re:The NumPy array: a structure for efficient numerical comp... by agibsonccc on Monday August 01, 2016 @07:05PM (#12206755)
We want to. Time/bandwidth is our problem right now. No one uses opencl in deep learning but we aren't against it if someone is willing to work on it.

Otherwise I'd have to pay someone a salary to do this..at the end of the day nd4j is part of what keeps us going.

Re:The NumPy array: a structure for efficient numerical comp... by doug1001 on Monday August 01, 2016 @11:41PM (#12207722)
your question caused me to spend a little time comparing the two (NumPy, which i know quite well, and nd4j, which i don't know at all).

so based on comparison, it's clear the nd4j uses the key structural features of numpy arrays--linear buffers, strides, bit-wise and integer indexing, etc.--and the coincidence in the two APIs is also very high (eg, comparing the methods for a 2D NumPy array versus those for a 2D nd4j array).

in my view, this is an impressive project, both w/r/t the work done to date and w/r/t the impact on java and scala open-source communities. In Scala land for instance, the most widely used matrix computation library is Breeze, which i've never found easy to use. nd4j, in particular, the Scala bindings, nd4s, seem to me to be a substantial improvement both in architecture and in the api.

Re:The NumPy array: a structure for efficient numerical comp... by nl on Monday August 01, 2016 @12:16PM (#12203593)
If you like NumPy, it might be worth taking a look at Dask[1]. Very similar API, but designed to support arrays larger than memory, as well as operations across a network cluster.

[1] http://dask.pydata.org/en/latest/

Re:The NumPy array: a structure for efficient numerical comp... by jdlyga on Monday August 01, 2016 @02:04AM (#12200478)
If only it was also available for C++. Armadillo is nice but it has a lot of important differences, and doesn't seem to support more than 3 dimensional matrices.
Re:The NumPy array: a structure for efficient numerical comp... by keypusher on Monday August 01, 2016 @03:13AM (#12200678)
There was some effort toward getting it in Boost. https://github.com/ndarray/Boost.NumPy
Re:The NumPy array: a structure for efficient numerical comp... by naveen99 on Monday August 01, 2016 @10:13AM (#12202497)
Arrayfire is c++ with Opencl, cuda, and cpu back ends. Limited to 4d. http://arrayfire.com/indexing-with-arrayfire/
Re:The NumPy array: a structure for efficient numerical comp... by lloda on Monday August 01, 2016 @07:05AM (#12201481)
Here is my Blitz++-inspired hobby project, supports arbitrary rank: https://github.com/lloda/ra-ra

I'm sure you can find more polished efforts if you look around.

Re:The NumPy array: a structure for efficient numerical comp... by srean on Monday August 01, 2016 @06:36AM (#12201338)
If you want multidimensional arrays in C++, then blitz++ is your friend.

There is Eigen but it wont do multidimensional stuff. Although there might be a few projects out there that wrap the multidimensional functionality around Eigen arrays.

Re:The NumPy array: a structure for efficient numerical comp... by kersny on Monday August 01, 2016 @10:04AM (#12202428)
One such multidimensional extension of Eigen is from Google and I believe was extracted from TensorFlow:

https://bitbucket.org/eigen/eigen/src/039efd86b75ce1b72befb7...

Re:The NumPy array: a structure for efficient numerical comp... by chombier on Monday August 01, 2016 @06:35AM (#12201327)
You might want to check out the Eigen library: http://eigen.tuxfamily.org/
Re:The NumPy array: a structure for efficient numerical comp... by mamcx on Monday August 01, 2016 @10:30AM (#12202613)
Also something like this but for .NET?
Re:The NumPy array: a structure for efficient numerical comp... by hatmatrix on Monday August 01, 2016 @07:09AM (#12201491)
It's a shame that there is no consolidated way to provide index labels to both columns and rows without wrapping this in another class (e.g., your own, or pandas). R's matrices and arrays are very convenient as they provide this functionality out-of-the-box.
Re:The NumPy array: a structure for efficient numerical comp... by cgearhart on Monday August 01, 2016 @12:39PM (#12203802)
I'm not terribly familiar with R, but I've used Pandas a bit. Could you accomplish what you're looking for with structured arrays or record arrays in Numpy?

http://docs.scipy.org/doc/numpy/user/basics.rec.html

Re:The NumPy array: a structure for efficient numerical comp... by hatmatrix on Monday August 01, 2016 @05:57PM (#12206369)
Record arrays only provide indexing by name along columns, unfortunately. So if you want index labels along both dimensions (in 2-D), Pandas is necessary. There is no good option for the N-D case that I can find (R's native array has this capability by default though).
Re:The NumPy array: a structure for efficient numerical comp... by noobermin on Monday August 01, 2016 @03:19AM (#12200699)
This type is my bread and butter. I breathe numpy arrays in my research.
Re:The NumPy array: a structure for efficient numerical comp... by jeroenjanssens on Monday August 01, 2016 @09:09AM (#12202058)
What an interesting juxtaposition of idioms :)
Re:The NumPy array: a structure for efficient numerical comp... by jostmey on Sunday July 31, 2016 @11:21PM (#12200027)
How about an efficient structure for storing "actual" arrays in python. Numpy is my friend.
Re:The NumPy array: a structure for efficient numerical comp... by Animats on Monday August 01, 2016 @12:08AM (#12200145)
Does PyPy have enough type inference smarts to figure out when it can use a homogeneous array? In CPython, everything, including every array entry, is a CObject. (Except strings and now, bytearrays. They're special.) That's why Python arrays of numbers are so slow.

NumPy's representation is very close to that of FORTRAN. This problem was mostly solved in the 1950s and 1960s, when FORTRAN DO-loop optimizations for arrays got good.

Re:The NumPy array: a structure for efficient numerical comp... by ludamad on Monday August 01, 2016 @01:12AM (#12200313)
http://doc.pypy.org/en/latest/release-1.8.0.html It does this since 1.8 it seems (mentioned under 'list specialization'). Probably 'type inference' isn't the right way to describe it. It will optimistically assume it's a homogeneous array and will have to recreate this array if the assumption is broken. This will be useful in the vast majority of cases, as mixed type arrays are not usually used in performance-sensitive code.
Re:The NumPy array: a structure for efficient numerical comp... by rlayton2 on Monday August 01, 2016 @01:25AM (#12200354)
Further, there is almost always a better data structure for any mixed-type list. So if you know your code will go through pypy, you can just use a different structure.
Re:The NumPy array: a structure for efficient numerical comp... by tgb on Monday August 01, 2016 @03:46AM (#12200774)
What kind of DO-loop optimizations were there?
Re:The NumPy array: a structure for efficient numerical comp... by Animats on Monday August 01, 2016 @03:52PM (#12205373)
The same ones we have now. Advancing through an array with pointers instead of computing the subscript each time. Advancing down a higher dimension of an array with a stride. Backus had those working in a FORTRAN compiler in the 1950s. If FORTRAN had been slower than assembly, nobody would have used it; machines were too slow and too expensive.
Re:The NumPy array: a structure for efficient numerical comp... by DasIch on Monday August 01, 2016 @10:58AM (#12202844)
You mean like the array[1] that's in the standard library?

[1]: https://docs.python.org/3/library/array.html

Re:The NumPy array: a structure for efficient numerical comp... by denfromufa on Monday August 01, 2016 @02:47PM (#12204869)
dynd is next generation numpy for language-agnostic usage: https://github.com/libdynd/libdynd