A Billion Rows per Second: Metaprogramming Python for Big Data [video]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by lcampbell on Friday September 27, 2013 @04:33PM (#6459344)

From my understanding of the video, the queries are generated by the analysts via a frontend-generated or hand-written SQL query. The SQL query is parsed by PostgreSQL which forwards it via FDW[1] to Multicorn[2]. Their custom data storage and processing backend implements the API expected by Multicorn (e.g., you implement the multicorn.ForeignDataWrapper interface); this is where they transform from the parsed, serialized SQL into their custom DSL (the metaprogramming bit) which compiles to LLVM.

[1] http://wiki.postgresql.org/wiki/Foreign_data_wrappers

[2] http://multicorn.org/

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by dialtone on Friday September 27, 2013 @06:48PM (#6460039)

Yeah, that's pretty much how it works. The frontend is anything that supports PostgreSQL as a database. Right now we use Tableau but we also used to have a custom WebUI on top of this service, it was very functional-inspired.

Unfortunately Ville is on vacation right now, otherwise he'd be glad to dive more into the details of how that piece worked.

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by vtuulos on Saturday September 28, 2013 @08:27AM (#6461647)

Author here: Fortunately HN works even in the Finnish countryside. I am happy to answer any questions.

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by iskander on Monday September 30, 2013 @01:36PM (#6470969)

Any interest in also trying Parakeet (https://github.com/iskandr/parakeet) for the backend? I'm curious to see how the performance would compare with Numba. I also have a semi-usable Builder API which constructs typed functions at a higher-level than llvmpy.

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by dev360 on Saturday September 28, 2013 @04:16PM (#6463022)

Very inspiring talk! Is it possible to deal with continuous data in those matrices or is it more oriented around discrete values?

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by vtuulos on Saturday September 28, 2013 @05:09PM (#6463168)

Thanks! Our approach supports both discrete and continuous values. It is mainly optimized for the use case where you want to aggregate continuous variables over discrete filters.

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by iskander on Friday September 27, 2013 @07:46PM (#6460249)

Why did you guys choose to compile through Numba rather than directly to LLVM or C?

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by vtuulos on Saturday September 28, 2013 @08:18AM (#6461631)

Numba was just the fastest way to get it working. LLVM(py) is very low-level. C is still an option but Numba made interfacing with a no-brainer.

[ Parent | on HN ]