Joining a billion rows 20x faster than Apache Spark

Re:Joining a billion rows 20x faster than Apache Spark by makapuf on Thursday February 09, 2017 @04:56PM (#13611394)

Processing data wouldn't be the problem with 2 socket xeons neither would it be putting 3 or 5 Hdd on a raid5. Getting the 32TB in, however, would take at least 8 hours at 10Gbps saturated, if your disks can write that fast.

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by mtanski on Thursday February 09, 2017 @06:18PM (#13611959)

10gigE seams fast, but in reality it's only 1.25GB/s in an ideal case. One enterprise PCIe SSD drive will saturate that. Or 5x of the old style 3.5 inch 7.2k RPM drives (you can fit 12 of these in a dense 1U case).

That why you see 40gigE or 56gigE used in HPC.

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by olavgg on Thursday February 09, 2017 @06:49PM (#13612119)

Its 25G, 50G or 100G today.

[ Parent | on HN ]