A Billion Rows per Second: Metaprogramming Python for Big Data [video]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by rtkwe on Friday September 27, 2013 @04:27PM (#6459316)

So in essence they do tons of pre-processing on their data. I wonder how long the pre-processing takes compared to the amount of speed gains it produced for them.

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by dialtone on Friday September 27, 2013 @06:45PM (#6460026)

Doesn't take so long actually, takes about 1 hour per day of data and every day we process about 10TB of uncompressed log files. The result of this can be stored and reused as many times as you need.

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by aborochoff on Friday September 27, 2013 @09:58PM (#6460568)

Interesting, you're processing ~500GB (10TB / 24 hour) of uncompressed loglines in 1 hour? Is the set up the same as the presentation?

[ Parent | on HN ]

Re:A Billion Rows per Second: Metaprogramming Python for Big... by lmm on Friday September 27, 2013 @07:24PM (#6460178)

How does this compare to something like spark/shark?

[ Parent | on HN ]