Re:1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
by lorenzhs on Wednesday January 25, 2017 @11:24AM
(#13482722)
A single machine doesn't have the required memory bandwidth to do this in the same time.
[ Parent
| on HN ]
Re:1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
by WhitneyLand on Wednesday January 25, 2017 @12:01PM
(#13483182)
Why not? Sounds like the initial data load and indexing are done up front. Once you get past that to run the benchmark it's not clear that a quad channel ddr4 system would be saturated.
Re:1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
by lorenzhs on Wednesday January 25, 2017 @04:56PM
(#13486609)
A current-generation Xeon E7 has a memory bandwidth of up to 102 GB/s. The article says 90, which is probably a realistic estimate of achievable throughput. But the data is 500GB. So the problem is not with loading it from disk (that can be done beforehand), but getting it from CPU to RAM fast enough.
Re:1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
by throwawayish on Thursday January 26, 2017 @07:57AM
(#13490929)
Inaccurate; the very point of columnar stores is that if I'm only interested in a column I only expend the memory bandwidth required for that very column and nothing else. Hence typical queries to a columnar store would never stream the whole 500 GB for processing.
Re:1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
by lorenzhs on Thursday January 26, 2017 @12:15PM
(#13493158)
Right, a columnar store would touch only a couple (1.1bn * three numerical columns) of those 500 GB. But the question was about Redis :)
The author has an overview of the benchmarks with various systems at http://tech.marksblogg.com/benchmarks.html – but one can't just compare rows as many factors vary (esp. hardware).