Metaprogramming Python For Big Data

Re:Metaprogramming Python For Big Data by vtuulos on Thursday December 12, 2013 @04:19AM (#6893584)

Author here: i just checked this morning - we have 100B+ rows in our deliroll matrices, hosted on a single machine.

I'm happy to answer any questions.

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by vtuulos on Thursday December 12, 2013 @10:26AM (#6894882)

Btw, if you would be interested in hacking Deliroll and other related things, feel free to contact me - my email is on the slides. We are hiring.

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by storrgie on Thursday December 12, 2013 @08:19AM (#6894229)

What did you use to make these slides?

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by vtuulos on Thursday December 12, 2013 @08:49AM (#6894349)

Reveal.js https://github.com/hakimel/reveal.js/

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by nly on Thursday December 12, 2013 @08:49AM (#6894351)

What data format are you using for your sparse matrices, and what's the INSERT performance like?

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by vtuulos on Thursday December 12, 2013 @10:12AM (#6894808)

Matrices are read-only. The set of matrices is updated once a day to include the latest data.

The data format is a collection of pre-aggregated row and column vectors, encoded with variable length integers and run-length encoding. I should give a separate presentation about this.

[ Parent | on HN ]

Re:Metaprogramming Python For Big Data by fixxer on Thursday December 12, 2013 @01:45PM (#6896209)

I should give a separate presentation about this.

Please do. Also, any change of releasing this?

[ Parent | on HN ]