Re:Metaprogramming Python For Big Data
by vtuulos on Thursday December 12, 2013 @10:12AM
(#6894808)
Matrices are read-only. The set of matrices is updated once a day to include the latest data.
The data format is a collection of pre-aggregated row and column vectors, encoded with variable length integers and run-length encoding. I should give a separate presentation about this.
Re:Metaprogramming Python For Big Data
by parrotdoxical on Thursday December 12, 2013 @09:37AM
(#6894601)
This is awesome. While it's great how performant the approach is, I also really dig how elegant the whole solution is -- using Postgres FDW with Numba is very pragmatic and clean, while at the same time potentially extensible to GPGPU. I might try and give this a go for some DSP stuff at some point.
Re:Metaprogramming Python For Big Data
by virmundi on Thursday December 12, 2013 @08:37AM
(#6894295)
It looks really neat. Isn't 660 GB not that much really? I grant that the slides say they've used an optimized binary for the storage, but how does this compare to pandas?
Re:Metaprogramming Python For Big Data
by vtuulos on Thursday December 12, 2013 @09:04AM
(#6894419)
660GB was just a small benchmark. The real thing uses more than a petabyte of raw data.
Pandas uses NumPy internally. You could use Deliroll as a replacement for NumPy in Pandas to get a nice interactive environment for amounts of data that can't be easily handled with plain NumPy.
Re:Metaprogramming Python For Big Data
by virmundi on Thursday December 12, 2013 @09:08AM
(#6894445)
This is interesting. My current project is a fraud detection system. We currently leverage Cascading/Hadoop. But I wanted to make sure the system is not Hadoop-centric. So I made a point of having the system be language agnostic. It looks like there might be a fit for this tool.
I passed the slides along to my team to see what they think. If they just impress upon the team that we need to store something other than just 0x0A delimited text files, I'll consider it a win.
Re:Metaprogramming Python For Big Data
by pwang on Thursday December 12, 2013 @10:40AM
(#6894980)
You can think of it like that. It's a Python -> Machine Code compiler based on LLVM, and it uses Numpy types (and Blaze types) to do type inference on numerical and data transformation functions.
Re:Metaprogramming Python For Big Data
by doug1001 on Friday December 13, 2013 @03:49AM
(#6899624)
really impressive work; i like how you guys (apparently) began with a blank page, and set aside at least a few stale assumptions that most consider inviolate principles in DW design--eg, denormalization, star/snowflake schema.
I'm happy to answer any questions.
The data format is a collection of pre-aggregated row and column vectors, encoded with variable length integers and run-length encoding. I should give a separate presentation about this.
Please do. Also, any change of releasing this?
I know continuum was working on a version of memmapped numpy (blaze, iirc) arrays which looked really interesting.
Pandas uses NumPy internally. You could use Deliroll as a replacement for NumPy in Pandas to get a nice interactive environment for amounts of data that can't be easily handled with plain NumPy.
I passed the slides along to my team to see what they think. If they just impress upon the team that we need to store something other than just 0x0A delimited text files, I'll consider it a win.
http://numba.pydata.org/