Re:TiSpark sits Spark SQL on top of a storage engine to answ...
by elvinyung on Friday August 04, 2017 @05:02PM
(#14932234)
Very nice! I actually read the FAST '17 paper a while back.
I'm curious if HopsFS has any support for partition-pruned secondary index scans? (If not, is it on the roadmap?) I feel like the lack of it in many current big-data systems is a major bottleneck to adoption for many kinds of workloads.
Re:TiSpark sits Spark SQL on top of a storage engine to answ...
by jamesblonde on Friday August 04, 2017 @05:46PM
(#14932595)
Do you mean Hive on Hops, I guess? HopsFS is a filesystem (drop-in replacement for HDFS with distributed metadata).
Partition-pruned index scans are key for high performance in distributed databases. Index scans and full-table scans hit all shards and kill performance. Hive supports partition-pruned index scans and push-down filtering for projections.
We have since adapted Hive SQL to move its metadata to the same DB as HopsFS (MySQL Cluster). We added foreign keys from Hive tables to the backing directories in HDFS. The result is strongly consistent metadata for Hive. That is, you can remove a backing directorty for a DB or table in HopsFS, and Hive's DB or table is automatically deleted. If you have ideas for next steps, we're all ears. It's all open-source: https://github.com/hopshadoop
See 22'40 in here: https://www.youtube.com/watch?v=mTRsrjH5WLI