News for hackers. Stuff that matters.
Re:TiSpark sits Spark SQL on top of a storage engine to answ... by jamesblonde on Friday August 04, 2017 @06:51AM (#14927735)
We have made a step towards unifying HDFS with a distributed SQL layer by doing something different- we put HDFS' metadata in MySQL Cluster (NDB) - an in-memory distributed DB that scales to 10s of nodes. NDB is not your father's MySQL. The result is HopsFS that scales to at least 16X HDFS on throughput (Usenix FAST 2017). See here: www.hops.io

We have since adapted Hive SQL to move its metadata to the same DB as HopsFS (MySQL Cluster). We added foreign keys from Hive tables to the backing directories in HDFS. The result is strongly consistent metadata for Hive. That is, you can remove a backing directorty for a DB or table in HopsFS, and Hive's DB or table is automatically deleted. If you have ideas for next steps, we're all ears. It's all open-source: https://github.com/hopshadoop

See 22'40 in here: https://www.youtube.com/watch?v=mTRsrjH5WLI

Re:TiSpark sits Spark SQL on top of a storage engine to answ... by elvinyung on Friday August 04, 2017 @05:02PM (#14932234)
Very nice! I actually read the FAST '17 paper a while back.

I'm curious if HopsFS has any support for partition-pruned secondary index scans? (If not, is it on the roadmap?) I feel like the lack of it in many current big-data systems is a major bottleneck to adoption for many kinds of workloads.

Re:TiSpark sits Spark SQL on top of a storage engine to answ... by jamesblonde on Friday August 04, 2017 @05:46PM (#14932595)
Do you mean Hive on Hops, I guess? HopsFS is a filesystem (drop-in replacement for HDFS with distributed metadata).

Partition-pruned index scans are key for high performance in distributed databases. Index scans and full-table scans hit all shards and kill performance. Hive supports partition-pruned index scans and push-down filtering for projections.