Metaprogramming Python For Big Data

Re:Metaprogramming Python For Big Data by Fede_V on Thursday December 12, 2013 @04:40AM (#6893634)

Awesome to see tech like numba being used like this. Are you guys going to write a more 'techy' version of how you accomplish all of this?

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by vtuulos on Thursday December 12, 2013 @04:46AM (#6893653)

Yes, definitely. We might be able to open-source some parts of the system eventually as well.

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by Fede_V on Thursday December 12, 2013 @06:50AM (#6893969)

Excellent. For sparse storage, do you use Scipy Sparse, or you have your own custom built solution?

I know continuum was working on a version of memmapped numpy (blaze, iirc) arrays which looked really interesting.

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by vtuulos on Thursday December 12, 2013 @08:54AM (#6894371)

I started with Scipy Sparse. I ditched it for a custom solution so I could support variable-length integers, run-length encoding etc.

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by pwang on Thursday December 12, 2013 @10:41AM (#6894983)

Good work Ville, let me know when you post those slides/writeup. :-)

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by vtuulos on Thursday December 12, 2013 @11:29AM (#6895289)

Thanks Peter! Let me know if you happen to visit SF. It would be nice to catch up.

[ Parent | on HN ]