News for hackers. Stuff that matters.

A comment on A Billion Rows per Second: Metaprogramming Python for Big Data [video]. Parent comment

Re:A Billion Rows per Second: Metaprogramming Python for Big... by dialtone on Friday September 27, 2013 @06:45PM (#6460026)
Doesn't take so long actually, takes about 1 hour per day of data and every day we process about 10TB of uncompressed log files. The result of this can be stored and reused as many times as you need.
[ Parent | on HN ]
Re:A Billion Rows per Second: Metaprogramming Python for Big... by aborochoff on Friday September 27, 2013 @09:58PM (#6460568)
Interesting, you're processing ~500GB (10TB / 24 hour) of uncompressed loglines in 1 hour? Is the set up the same as the presentation?
[ Parent | on HN ]
Re:A Billion Rows per Second: Metaprogramming Python for Big... by lmm on Friday September 27, 2013 @07:24PM (#6460178)
How does this compare to something like spark/shark?
[ Parent | on HN ]