Re:BlazingDB Origins, oh and we just raised $2.9M from Nvidi...
by ruw1090 on Friday December 01, 2017 @12:47AM
(#15822187)
What is the benefit of shipping data to the GPU for execution if the data is on S3 or HDFS? Won't most of the cost of the query be I/O?
[ Parent
| on HN ]
Re:BlazingDB Origins, oh and we just raised $2.9M from Nvidi...
by felipe_aramburu on Friday December 01, 2017 @03:33AM
(#15822705)
Sure the very first time you run a query. But with multi tiered caching the data you frequently access sits closer and closer to the gpus so that alleviates that bottlekneck over time to an extent. Also what is a fantastic way of improving i/o? Compression and decompression. Our own file format compresses and decompresses using the GPU. We are working on doing the same for some of the Parquet decompression steps. I/O is almost always your main concern here, but you can improve upon it greatly by leverage processes that might not have been computationally feasible before.